[jira] [Commented] (HADOOP-15338) Support Java 11 LTS in Hadoop

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902490#comment-16902490
 ] 

Matt Foley commented on HADOOP-15338:
-

MCOMPILER-342 shows fixed for the compiler plugin in version 3.8.0.  But the 
jira notes other parts of the maven system (specifically maven-plugin-plugin) 
are not there yet.

> Support Java 11 LTS in Hadoop
> -
>
> Key: HADOOP-15338
> URL: https://issues.apache.org/jira/browse/HADOOP-15338
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
>
> Oracle JDK 8 will be EoL during January 2019, and RedHat will end support for 
> OpenJDK 8 in June 2023 ([https://access.redhat.com/articles/1299013]), so we 
> need to support Java 11 LTS at least before June 2023.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-16166.
-
Resolution: Fixed

> TestRawLocalFileSystemContract fails with build Docker container running on 
> Mac
> ---
>
> Key: HADOOP-16166
> URL: https://issues.apache.org/jira/browse/HADOOP-16166
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Matt Foley
>Assignee: Matt Foley
>Priority: Minor
>
> The Mac has a case-insensitive filesystem.  When using the recommended build 
> Docker container via `start-build-env.sh`, the container attaches to the Mac 
> FS to share the local git repository for Hadoop.  Which is very nice and 
> convenient.
> This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
> test case (which is inherited from FileSystemContractBaseTest) should be 
> skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
> because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
> does not take into account the possibility of a Linux OS mounting a MacOS 
> filesystem.
> The fix would extend 
> TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this 
> case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791901#comment-16791901
 ] 

Matt Foley commented on HADOOP-16109:
-

Thanks very much [~ste...@apache.org]!

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HADOOP-16109-branch-3.1-003.patch
>
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>  at java.lang.Thread.run(Thread.java:748)
> {code}
> The following example program generate the same root behavior (sans finding a 
> Parquet file that happens to trigger this condition) by purposely reading 
> past the already active readahead range on any file >= 1029 bytes in size.. 
> {code:java}
> final Configuration conf = new Configuration();
> conf.set("fs.s3a.readahead.range", "1K");
> conf.set("fs.s3a.experimental.input.fadvise", "random");
> final FileSystem fs = FileSystem.get(path.toUri(), conf);
> // forward seek reading across readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
> final byte[] temp = new byte[5];
> in.readByte();
> in.readFully(1023, temp); // <-- works
> }
> // forward seek reading from end of readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
>  final byte[] temp = new byte[5];
>  in.readByte();
>  in.readFully(1024, temp); // <-- throws EOFException
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788223#comment-16788223
 ] 

Matt Foley edited comment on HADOOP-16166 at 3/8/19 8:17 PM:
-

Submitted fix as PR# 580
Included similar fix for Windows.


was (Author: mattf):
Submitted fix as PR# 580

> TestRawLocalFileSystemContract fails with build Docker container running on 
> Mac
> ---
>
> Key: HADOOP-16166
> URL: https://issues.apache.org/jira/browse/HADOOP-16166
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Matt Foley
>Assignee: Matt Foley
>Priority: Minor
>
> The Mac has a case-insensitive filesystem.  When using the recommended build 
> Docker container via `start-build-env.sh`, the container attaches to the Mac 
> FS to share the local git repository for Hadoop.  Which is very nice and 
> convenient.
> This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
> test case (which is inherited from FileSystemContractBaseTest) should be 
> skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
> because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
> does not take into account the possibility of a Linux OS mounting a MacOS 
> filesystem.
> The fix would extend 
> TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this 
> case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reassigned HADOOP-16166:
---

Assignee: Matt Foley

> TestRawLocalFileSystemContract fails with build Docker container running on 
> Mac
> ---
>
> Key: HADOOP-16166
> URL: https://issues.apache.org/jira/browse/HADOOP-16166
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Matt Foley
>Assignee: Matt Foley
>Priority: Minor
>
> The Mac has a case-insensitive filesystem.  When using the recommended build 
> Docker container via `start-build-env.sh`, the container attaches to the Mac 
> FS to share the local git repository for Hadoop.  Which is very nice and 
> convenient.
> This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
> test case (which is inherited from FileSystemContractBaseTest) should be 
> skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
> because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
> does not take into account the possibility of a Linux OS mounting a MacOS 
> filesystem.
> The fix would extend 
> TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this 
> case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788223#comment-16788223
 ] 

Matt Foley commented on HADOOP-16166:
-

Submitted fix as PR# 580

> TestRawLocalFileSystemContract fails with build Docker container running on 
> Mac
> ---
>
> Key: HADOOP-16166
> URL: https://issues.apache.org/jira/browse/HADOOP-16166
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Matt Foley
>Priority: Minor
>
> The Mac has a case-insensitive filesystem.  When using the recommended build 
> Docker container via `start-build-env.sh`, the container attaches to the Mac 
> FS to share the local git repository for Hadoop.  Which is very nice and 
> convenient.
> This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
> test case (which is inherited from FileSystemContractBaseTest) should be 
> skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
> because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
> does not take into account the possibility of a Linux OS mounting a MacOS 
> filesystem.
> The fix would extend 
> TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this 
> case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780826#comment-16780826
 ] 

Matt Foley edited comment on HADOOP-16109 at 3/7/19 12:53 AM:
--

Hi [~ste...@apache.org], one of my colleagues, Shruti Gumma, has a proposed fix 
which I'll help him post here today.

Edit: Shruti is unable to test on S3, so requests you proceed with your fix.


was (Author: mattf):
Hi [~ste...@apache.org], one of my colleagues, Shruti Gumma, has a proposed fix 
which I'll help him post here today.

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>  at java.lang.Thread.run(Thread.java:748)
> {code}
> The following example program generate the same root behavior (sans finding a 
> Parquet file that happens to trigger this condition) by purposely reading 
> past the already active readahead range on any file >= 1029 bytes in size.. 
> {code:java}
> final Configuration conf = new Configuration();
> conf.set("fs.s3a.readahead.range", "1K");
> conf.set("fs.s3a.experimental.input.fadvise", "random");
> final FileSystem fs = FileSystem.get(path.toUri(), conf);
> // forward seek reading across readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
> final byte[] temp = new byte[5];
> in.readByte();
> in.readFully(1023, temp); // <-- works
> }
> // forward seek reading from end of readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
>  final byte[] temp = new byte[5];
>  in.readByte();
>  in.readFully(1024, temp); // <-- throws EOFException
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783920#comment-16783920
 ] 

Matt Foley commented on HADOOP-16166:
-

Yes, which is exactly what testFilesystemIsCaseSensitive() tests :D That would 
get a little tautological...
Probably best to stick with current model of identifying which systems we 
expect to be case-(in)sensitive, or not.
I'm putting together a patch using `df .` which returns "osxfs" in the first 
field, for the case of interest.

> TestRawLocalFileSystemContract fails with build Docker container running on 
> Mac
> ---
>
> Key: HADOOP-16166
> URL: https://issues.apache.org/jira/browse/HADOOP-16166
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Matt Foley
>Priority: Minor
>
> The Mac has a case-insensitive filesystem.  When using the recommended build 
> Docker container via `start-build-env.sh`, the container attaches to the Mac 
> FS to share the local git repository for Hadoop.  Which is very nice and 
> convenient.
> This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
> test case (which is inherited from FileSystemContractBaseTest) should be 
> skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
> because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
> does not take into account the possibility of a Linux OS mounting a MacOS 
> filesystem.
> The fix would extend 
> TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this 
> case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-16166:

Description: 
The Mac has a case-insensitive filesystem.  When using the recommended build 
Docker container via `start-build-env.sh`, the container attaches to the Mac FS 
to share the local git repository for Hadoop.  Which is very nice and 
convenient.

This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
test case (which is inherited from FileSystemContractBaseTest) should be 
skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
does not take into account the possibility of a Linux OS mounting a MacOS 
filesystem.

The fix would extend TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
to recognize this case.

  was:
The Mac has a case-insensitive filesystem.  When using the recommended build 
Docker container via `start-build-env.sh`, the container attaches to the Mac FS 
to share the local git repository for Hadoop.  Which is very nice and 
convenient.

This means the TestRawLocalFileSystemContract::testFilesystemIsCaseSensitive() 
test case (which is inherited from FileSystemContractBaseTest) should be 
skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
because @Override TestRawLocalFileSystemContract::filesystemIsCaseSensitive() 
does not take into account the possibility of a Linux OS mounting a MacOS 
filesystem.

The fix would extend 
TestRawLocalFileSystemContract::filesystemIsCaseSensitive() to recognize this 
case.


> TestRawLocalFileSystemContract fails with build Docker container running on 
> Mac
> ---
>
> Key: HADOOP-16166
> URL: https://issues.apache.org/jira/browse/HADOOP-16166
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Matt Foley
>Priority: Minor
>
> The Mac has a case-insensitive filesystem.  When using the recommended build 
> Docker container via `start-build-env.sh`, the container attaches to the Mac 
> FS to share the local git repository for Hadoop.  Which is very nice and 
> convenient.
> This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() 
> test case (which is inherited from FileSystemContractBaseTest) should be 
> skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
> because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() 
> does not take into account the possibility of a Linux OS mounting a MacOS 
> filesystem.
> The fix would extend 
> TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this 
> case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac

[Matt Foley (JIRA)]

Matt Foley created HADOOP-16166:
---

 Summary: TestRawLocalFileSystemContract fails with build Docker 
container running on Mac
 Key: HADOOP-16166
 URL: https://issues.apache.org/jira/browse/HADOOP-16166
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 3.3.0
Reporter: Matt Foley


The Mac has a case-insensitive filesystem.  When using the recommended build 
Docker container via `start-build-env.sh`, the container attaches to the Mac FS 
to share the local git repository for Hadoop.  Which is very nice and 
convenient.

This means the TestRawLocalFileSystemContract::testFilesystemIsCaseSensitive() 
test case (which is inherited from FileSystemContractBaseTest) should be 
skipped.  It fails to be skipped, and therefore throws a Unit Test failure, 
because @Override TestRawLocalFileSystemContract::filesystemIsCaseSensitive() 
does not take into account the possibility of a Linux OS mounting a MacOS 
filesystem.

The fix would extend 
TestRawLocalFileSystemContract::filesystemIsCaseSensitive() to recognize this 
case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137
 ] 

Matt Foley edited comment on HADOOP-16109 at 3/1/19 12:49 AM:
--

Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

{{&& diff < forwardSeekLimit;}} instead of {{<=}}

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
{{remainingInCurrentRequest}}, as being a problem, but it seems to me that 
should be okay; the problem documented so far is *seeking* past 
{{remainingInCurrentRequest}} (specifically to exactly the end of 
CurrentRequest, which is incorrectly guarded by the above L264 inequality) and 
then not doing a stream close, which causes the problem.  Do you know if, say, 
seeking to a few bytes before the end of CurrentRequest, then reading past it 
(when the S3 file does indeed have more to read), also causes an EOF, or does 
the stream machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 


was (Author: mattf):
Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

{{ && diff < forwardSeekLimit; }} instead of {{ <= }}

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
{{remainingInCurrentRequest}}, as being a problem, but it seems to me that 
should be okay; the problem documented so far is *seeking* past 
{{remainingInCurrentRequest}} (specifically to exactly the end of 
CurrentRequest, which is incorrectly guarded by the above L264 inequality) and 
then not doing a stream close, which causes the problem.  Do you know if, say, 
seeking to a few bytes before the end of CurrentRequest, then reading past it 
(when the S3 file does indeed have more to read), also causes an EOF, or does 
the stream machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at 

[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137
 ] 

Matt Foley edited comment on HADOOP-16109 at 3/1/19 12:50 AM:
--

Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

{{&& diff < forwardSeekLimit;  // instead of <=}}

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
{{remainingInCurrentRequest}}, as being a problem, but it seems to me that 
should be okay; the problem documented so far is *seeking* past 
{{remainingInCurrentRequest}} (specifically to exactly the end of 
CurrentRequest, which is incorrectly guarded by the above L264 inequality) and 
then not doing a stream close, which causes the problem.  Do you know if, say, 
seeking to a few bytes before the end of CurrentRequest, then reading past it 
(when the S3 file does indeed have more to read), also causes an EOF, or does 
the stream machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 


was (Author: mattf):
Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

{{&& diff < forwardSeekLimit;}} instead of {{<=}}

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
{{remainingInCurrentRequest}}, as being a problem, but it seems to me that 
should be okay; the problem documented so far is *seeking* past 
{{remainingInCurrentRequest}} (specifically to exactly the end of 
CurrentRequest, which is incorrectly guarded by the above L264 inequality) and 
then not doing a stream close, which causes the problem.  Do you know if, say, 
seeking to a few bytes before the end of CurrentRequest, then reading past it 
(when the S3 file does indeed have more to read), also causes an EOF, or does 
the stream machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at 

[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137
 ] 

Matt Foley edited comment on HADOOP-16109 at 3/1/19 12:48 AM:
--

Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

{{ && diff < forwardSeekLimit; }} instead of {{ <= }}

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
{{remainingInCurrentRequest}}, as being a problem, but it seems to me that 
should be okay; the problem documented so far is *seeking* past 
{{remainingInCurrentRequest}} (specifically to exactly the end of 
CurrentRequest, which is incorrectly guarded by the above L264 inequality) and 
then not doing a stream close, which causes the problem.  Do you know if, say, 
seeking to a few bytes before the end of CurrentRequest, then reading past it 
(when the S3 file does indeed have more to read), also causes an EOF, or does 
the stream machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 


was (Author: mattf):
Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

`&& diff < forwardSeekLimit;` instead of `<=`

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
`remainingInCurrentRequest`, as being a problem, but it seems to me that should 
be okay; the problem documented so far is *seeking* past 
`remainingInCurrentRequest` (specifically to exactly the end of CurrentRequest, 
which is incorrectly guarded by the above L264 inequality) and then not doing a 
stream close, which causes the problem.  Do you know if, say, seeking to a few 
bytes before the end of CurrentRequest, then reading past it (when the S3 file 
does indeed have more to read), also causes an EOF, or does the stream 
machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at 

[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137
 ] 

Matt Foley commented on HADOOP-16109:
-

Yes, I'm thinking at 
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264]
 we need

`&& diff < forwardSeekLimit;` instead of `<=`

What do you think?

The big question I have is, some of the text description talks about "reading 
past the already active readahead range", i.e., past 
`remainingInCurrentRequest`, as being a problem, but it seems to me that should 
be okay; the problem documented so far is *seeking* past 
`remainingInCurrentRequest` (specifically to exactly the end of CurrentRequest, 
which is incorrectly guarded by the above L264 inequality) and then not doing a 
stream close, which causes the problem.  Do you know if, say, seeking to a few 
bytes before the end of CurrentRequest, then reading past it (when the S3 file 
does indeed have more to read), also causes an EOF, or does the stream 
machinery handle that case correctly?

I'm putting together a test platform so I can answer such questions myself, but 
it will take me a few hours; I haven't worked in s3a before.

 

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>  at java.lang.Thread.run(Thread.java:748)
> {code}
> The following example program generate the same root behavior (sans finding a 
> Parquet file that happens to trigger this condition) by purposely reading 
> past the already active readahead range on any file >= 1029 bytes in size.. 
> {code:java}
> final Configuration conf = new Configuration();
> conf.set("fs.s3a.readahead.range", "1K");
> conf.set("fs.s3a.experimental.input.fadvise", "random");
> final FileSystem fs = FileSystem.get(path.toUri(), conf);
> // forward seek reading across readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
> final byte[] temp = new byte[5];
> in.readByte();
> in.readFully(1023, temp); // <-- works
> }
> // forward seek reading from end of readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
>  final byte[] temp = new byte[5];
>  in.readByte();
>  in.readFully(1024, temp); // <-- throws EOFException
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: 

[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780826#comment-16780826
 ] 

Matt Foley commented on HADOOP-16109:
-

Hi [~ste...@apache.org], one of my colleagues, Shruti Gumma, has a proposed fix 
which I'll help him post here today.

> Parquet reading S3AFileSystem causes EOF
> 
>
> Key: HADOOP-16109
> URL: https://issues.apache.org/jira/browse/HADOOP-16109
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Dave Christianson
>Assignee: Steve Loughran
>Priority: Blocker
>
> When using S3AFileSystem to read Parquet files a specific set of 
> circumstances causes an  EOFException that is not thrown when reading the 
> same file from local disk
> Note this has only been observed under specific circumstances:
>   - when the reader is doing a projection (will cause it to do a seek 
> backwards and put the filesystem into random mode)
>  - when the file is larger than the readahead buffer size
>  - when the seek behavior of the Parquet reader causes the reader to seek 
> towards the end of the current input stream without reopening, such that the 
> next read on the currently open stream will read past the end of the 
> currently open stream.
> Exception from Parquet reader is as follows:
> {code}
> Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left 
> to read
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127)
>  at 
> org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174)
>  at 
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
>  at 
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
>  at 
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206)
>  at 
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199)
>  at 
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190)
>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>  at java.lang.Thread.run(Thread.java:748)
> {code}
> The following example program generate the same root behavior (sans finding a 
> Parquet file that happens to trigger this condition) by purposely reading 
> past the already active readahead range on any file >= 1029 bytes in size.. 
> {code:java}
> final Configuration conf = new Configuration();
> conf.set("fs.s3a.readahead.range", "1K");
> conf.set("fs.s3a.experimental.input.fadvise", "random");
> final FileSystem fs = FileSystem.get(path.toUri(), conf);
> // forward seek reading across readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
> final byte[] temp = new byte[5];
> in.readByte();
> in.readFully(1023, temp); // <-- works
> }
> // forward seek reading from end of readahead boundary
> try (FSDataInputStream in = fs.open(path)) {
>  final byte[] temp = new byte[5];
>  in.readByte();
>  in.readFully(1024, temp); // <-- throws EOFException
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-9359) Add Windows build and unit test to test-patch pre-commit testing

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reassigned HADOOP-9359:
--

Assignee: (was: Matt Foley)

> Add Windows build and unit test to test-patch pre-commit testing
> 
>
> Key: HADOOP-9359
> URL: https://issues.apache.org/jira/browse/HADOOP-9359
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: native
>Reporter: Matt Foley
>Priority: Major
>
> The "test-patch" utility is triggered by "Patch Available" state in Jira, and 
> runs nine different sets of builds, tests, and static analysis tools.  
> Currently only the Linux environment is tested.  Need to add tests for Java 
> build under Windows, and unit test execution under Windows.
> At this time, the community has decided that "-1" on these new additional 
> tests shall not block commits to the code base.  However, contributors and 
> code reviewers are encouraged to utilize the information provided by these 
> tests to help keep Hadoop cross-platform compatible.  Modify 
> http://wiki.apache.org/hadoop/HowToContribute to document this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-9359) Add Windows build and unit test to test-patch pre-commit testing

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-9359.

Resolution: Won't Fix

Closed due to evident insufficient interest.

> Add Windows build and unit test to test-patch pre-commit testing
> 
>
> Key: HADOOP-9359
> URL: https://issues.apache.org/jira/browse/HADOOP-9359
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: native
>Reporter: Matt Foley
>Priority: Major
>
> The "test-patch" utility is triggered by "Patch Available" state in Jira, and 
> runs nine different sets of builds, tests, and static analysis tools.  
> Currently only the Linux environment is tested.  Need to add tests for Java 
> build under Windows, and unit test execution under Windows.
> At this time, the community has decided that "-1" on these new additional 
> tests shall not block commits to the code base.  However, contributors and 
> code reviewers are encouraged to utilize the information provided by these 
> tests to help keep Hadoop cross-platform compatible.  Modify 
> http://wiki.apache.org/hadoop/HowToContribute to document this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-7158) Reduce RPC packet size for homogeneous arrays, such as the array responses to listStatus() and getBlockLocations()

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reassigned HADOOP-7158:
--

Assignee: (was: Matt Foley)

> Reduce RPC packet size for homogeneous arrays, such as the array responses to 
> listStatus() and getBlockLocations()
> --
>
> Key: HADOOP-7158
> URL: https://issues.apache.org/jira/browse/HADOOP-7158
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.22.0
>Reporter: Matt Foley
>Priority: Major
>
> While commenting on HADOOP-6949, which proposes a big improvement in the RPC 
> wire format for arrays of primitives, Konstantin Shvachko said:
> "Can/should we extend this to arrays of non-primitive types? This should 
> benefit return types for calls like listStatus() and getBlockLocations() on a 
> large directory."
> The improvement for primitive arrays is based on not type-labeling every 
> element in the array, so the array in question must be strictly homogenous; 
> it cannot have subtypes of the assignable type.  For instance, it could not 
> be applied to heartbeat responses of DatanodeCommand[], whose array elements 
> carry subtypes of DatanodeCommand, each of which must be type-labeled 
> independently.  However, as Konstantin points out, it could really help 
> lengthy response arrays for things like listStatus() and getBlockLocations().
> I will attach a prototype implementation to this Jira, for discussion.  
> However, since it can't be automatically applied to all arrays passing 
> through RPC, I'll just providing the wrapper type.  By using it, a caller is 
> asserting that the array is strictly homogeneous in the above sense.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-7809) Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote job submission

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-7809.

Resolution: Won't Do

Aged out.

> Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote 
> job submission
> ---
>
> Key: HADOOP-7809
> URL: https://issues.apache.org/jira/browse/HADOOP-7809
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: contrib/cloud
>Reporter: Joydeep Sen Sarma
>Priority: Major
> Attachments: ASF.LICENSE.NOT.GRANTED--hadoop-5839.2.patch
>
>
> The fix for HADOOP-5839 was committed to 0.21 more than a year ago.  This bug 
> is to backport the change (which is only 14 lines) to branch-0.20-security.
> ===
> Original description:
> i would very much like the option of submitting jobs from a workstation 
> outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a 
> socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private 
> hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the 
> generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to 
> internal/private IP addresses within ec2 (so traffic is optimally routed).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-7809) Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote job submission

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reassigned HADOOP-7809:
--

Assignee: (was: Matt Foley)

> Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote 
> job submission
> ---
>
> Key: HADOOP-7809
> URL: https://issues.apache.org/jira/browse/HADOOP-7809
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: contrib/cloud
>Reporter: Joydeep Sen Sarma
>Priority: Major
> Attachments: ASF.LICENSE.NOT.GRANTED--hadoop-5839.2.patch
>
>
> The fix for HADOOP-5839 was committed to 0.21 more than a year ago.  This bug 
> is to backport the change (which is only 14 lines) to branch-0.20-security.
> ===
> Original description:
> i would very much like the option of submitting jobs from a workstation 
> outside ec2 to a hadoop cluster in ec2. This has been explored here:
> http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html
> the net result of this is that we can make this work (along with using a 
> socks proxy) with a couple of changes in the ec2 scripts:
> a) use public 'hostname' for fs.default.name setting (instead of the private 
> hostname being used currently)
> b) mark hadoop.rpc.socket.factory.class.default as final variable in the 
> generated hadoop-site.xml (that applies to server side)
> #a has no downside as far as i can tell since public hostnames resolve to 
> internal/private IP addresses within ec2 (so traffic is optimally routed).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Description: 
In branch-2.8 and later, the patches for various child and related bugs listed 
in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
"commons-httpclient" from Hadoop and its sub-projects (except for 
hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)
In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
inherited from hadoop-project/pom.xml also needs to be added, so that is in the 
branch-2.8 version of the patch.

Other projects with undeclared transitive dependencies on commons-httpclient, 
previously provided via hadoop-common or hadoop-client, may find this to be an 
incompatible change.  Of course that also means such project is exposed to the 
commons-httpclient CVE, and needs to be fixed for that reason as well.


  was:
In branch-2.8 and later, the patches for various child and related bugs listed 
in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
"commons-httpclient" from Hadoop and its sub-projects (except for 
hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)
In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
inherited from hadoop-project/pom.xml also needs to be added, so that is in the 
branch-2.8 version of the patch.



> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Fix For: 2.8.0
>
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.
> Other projects with undeclared transitive dependencies on commons-httpclient, 
> previously provided via hadoop-common or hadoop-client, may find this to be 
> an incompatible change.  Of course that also means such project is exposed to 
> the commons-httpclient CVE, and needs to be fixed for that reason as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412097#comment-15412097
 ] 

Matt Foley commented on HADOOP-13382:
-

[~steve_l]: sorry.  Fixed in 2.8.0.
[~gsaha]: It is true that other projects with undeclared transitive 
dependencies on commons-httpclient, previously provided via hadoop-common or 
hadoop-client, may find this to be an incompatible change.  Of course that also 
means such project is exposed to the commons-httpclient CVE, and needs to be 
fixed for that reason as well.  Will update the Description to note this.  
Thanks for setting the appropriate flags in the jira.

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Fix For: 2.8.0
>
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Fix Version/s: 2.8.0

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Fix For: 2.8.0
>
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks, [~cnauroth] and [~steve_l].  Committed as:
* trunk - 12aa184479675d6c9bd
* branch-2 - ea10e1384ff65e27521
* branch-2.8 - c96cb3fd48925b3eb2c

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384907#comment-15384907
 ] 

Matt Foley edited comment on HADOOP-13382 at 7/19/16 10:15 PM:
---

Response to the Hadoop QA robot complaint about no new unit tests: This patch 
seeks to produce no functional change in the behavior of the code, therefore 
there are no new unit tests needed.  There are no new negative tests needed 
either, because if the patch breaks anything, it will be a gross breakage of 
hadoop-openstack.  Since all existing unit tests continue to work correctly, 
that's sufficient.


was (Author: mattf):
Response to the Hadoop QA complaint: This patch seeks to produce no functional 
change in the behavior of the code, therefore there are no new unit tests 
needed.  There are no new negative tests needed either, because if the patch 
breaks anything, it will be a gross breakage of hadoop-openstack.  Since all 
existing unit tests continue to work correctly, that's sufficient.

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384907#comment-15384907
 ] 

Matt Foley commented on HADOOP-13382:
-

Response to the Hadoop QA complaint: This patch seeks to produce no functional 
change in the behavior of the code, therefore there are no new unit tests 
needed.  There are no new negative tests needed either, because if the patch 
breaks anything, it will be a gross breakage of hadoop-openstack.  Since all 
existing unit tests continue to work correctly, that's sufficient.

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Description: 
In branch-2.8 and later, the patches for various child and related bugs listed 
in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
"commons-httpclient" from Hadoop and its sub-projects (except for 
hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)
In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
inherited from hadoop-project/pom.xml also needs to be added, so that is in the 
branch-2.8 version of the patch.


  was:
The patches for various child and related bugs listed in HADOOP-10105, most 
recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and 
HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
* (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)
In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
inherited from hadoop-project/pom.xml also needs to be added, so that is in the 
branch-2.8 version of the patch.



> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Fix For: 2.8.0
>
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Fix Version/s: 2.8.0

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Fix For: 2.8.0
>
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> In branch-2.8 and later, the patches for various child and related bugs 
> listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, 
> HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of 
> "commons-httpclient" from Hadoop and its sub-projects (except for 
> hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-11614) Remove httpclient dependency from hadoop-openstack

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-11614:

Description: Remove httpclient dependency from hadoop-openstack and its 
pom.xml file.  (was: Remove httpclient dependency from hadoop-openstack.)

> Remove httpclient dependency from hadoop-openstack
> --
>
> Key: HADOOP-11614
> URL: https://issues.apache.org/jira/browse/HADOOP-11614
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-11614-002.patch, HADOOP-11614.patch
>
>
> Remove httpclient dependency from hadoop-openstack and its pom.xml file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Affects Version/s: 2.8.0
 Target Version/s: 2.8.0
   Status: Patch Available  (was: In Progress)

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.8.0
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, 
> and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
> sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Attachment: HADOOP-13382.000.patch
HADOOP-13382-branch-2.000.patch

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.000.patch, 
> HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, 
> and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
> sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380340#comment-15380340
 ] 

Matt Foley edited comment on HADOOP-13382 at 7/16/16 1:21 AM:
--

Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and 
HADOOP-12552. 

Branch-2 and trunk need exactly the same patch.


was (Author: mattf):
Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and 
HADOOP-12552. 

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.8.000.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, 
> and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
> sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Attachment: HADOOP-13382-branch-2.8.000.patch

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.8.000.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, 
> and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
> sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380340#comment-15380340
 ] 

Matt Foley edited comment on HADOOP-13382 at 7/16/16 12:57 AM:
---

Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and 
HADOOP-12552. 


was (Author: mattf):
Proposed for branch-2.8, to go along with HADOOP-11613 and HADOOP-12711. 

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, 
> and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
> sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Description: 
The patches for various child and related bugs listed in HADOOP-10105, most 
recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and 
HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
* (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)
In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
inherited from hadoop-project/pom.xml also needs to be added, so that is in the 
branch-2.8 version of the patch.


  was:
The patches for various child and related bugs listed in HADOOP-10105, most 
recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
(except for hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-common-project/hadoop-common/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)



> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, 
> and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its 
> sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml 
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)
> In 2.8, this was mostly done by HADOOP-12552, but the version info formerly 
> inherited from hadoop-project/pom.xml also needs to be added, so that is in 
> the branch-2.8 version of the patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380340#comment-15380340
 ] 

Matt Foley edited comment on HADOOP-13382 at 7/16/16 12:27 AM:
---

Proposed for branch-2.8, to go along with HADOOP-11613 and HADOOP-12711. 


was (Author: mattf):
Proposed for branch-2.8

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Attachment: (was: HADOOP-13382-branch-2.8.000.patch)

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Attachment: HADOOP-13382-branch-2.8.000.patch

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.8.000.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Attachment: (was: HADOOP-13382.patch)

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382-branch-2.8.000.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-13382:

Attachment: HADOOP-13382.patch

Proposed for branch-2.8

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-13382.patch
>
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Work started] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-13382 started by Matt Foley.
---
> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reassigned HADOOP-13382:
---

Assignee: Matt Foley

> remove unneeded commons-httpclient dependencies from POM files in Hadoop and 
> sub-projects
> -
>
> Key: HADOOP-13382
> URL: https://issues.apache.org/jira/browse/HADOOP-13382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> The patches for various child and related bugs listed in HADOOP-10105, most 
> recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
> eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
> (except for hadoop-tools/hadoop-openstack; see HADOOP-11614).
> However, after incorporating these patches, "commons-httpclient" is still 
> listed as a dependency in these POM files:
> * hadoop-project/pom.xml
> * hadoop-common-project/hadoop-common/pom.xml
> * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml
> We wish to remove these, but since commons-httpclient is still used in many 
> files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
> * hadoop-tools/hadoop-openstack/pom.xml
> (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
> removed from hadoop-openstack.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380332#comment-15380332
 ] 

Matt Foley commented on HADOOP-12710:
-

This jira is marked fixed in 2.9.0, which is true.  However, it is worth noting 
that the affected file, 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServerLogs.java,
 is new in 2.9.0.  That is, it was not introduced to branch-2 until after 
branch-2.8 was forked off, so the issue does not exist in 2.8.0 or earlier.

> Remove dependency on commons-httpclient for TestHttpServerLogs
> --
>
> Key: HADOOP-12710
> URL: https://issues.apache.org/jira/browse/HADOOP-12710
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.9.0
>
> Attachments: HADOOP-12710.001.patch
>
>
> Commons-httpclient has long been EOL. Critically, it has several security 
> vulnerabilities: CVE-2012-5783 
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783.
> I saw a recent commit that depends on commons-httpclient for 
> TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency 
> with httpclient APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12710:

Comment: was deleted

(was: -Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which 
is not yet released.-
Apologies, my mistake.  Only went into trunk.)

> Remove dependency on commons-httpclient for TestHttpServerLogs
> --
>
> Key: HADOOP-12710
> URL: https://issues.apache.org/jira/browse/HADOOP-12710
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.9.0
>
> Attachments: HADOOP-12710.001.patch
>
>
> Commons-httpclient has long been EOL. Critically, it has several security 
> vulnerabilities: CVE-2012-5783 
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783.
> I saw a recent commit that depends on commons-httpclient for 
> TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency 
> with httpclient APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380304#comment-15380304
 ] 

Matt Foley edited comment on HADOOP-12710 at 7/15/16 11:44 PM:
---

-Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is 
not yet released.-
Apologies, my mistake.  Only went into trunk.


was (Author: mattf):
Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is not 
yet released.

> Remove dependency on commons-httpclient for TestHttpServerLogs
> --
>
> Key: HADOOP-12710
> URL: https://issues.apache.org/jira/browse/HADOOP-12710
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.9.0
>
> Attachments: HADOOP-12710.001.patch
>
>
> Commons-httpclient has long been EOL. Critically, it has several security 
> vulnerabilities: CVE-2012-5783 
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783.
> I saw a recent commit that depends on commons-httpclient for 
> TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency 
> with httpclient APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12710:

Fix Version/s: (was: 2.8.0)
   2.9.0

> Remove dependency on commons-httpclient for TestHttpServerLogs
> --
>
> Key: HADOOP-12710
> URL: https://issues.apache.org/jira/browse/HADOOP-12710
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.9.0
>
> Attachments: HADOOP-12710.001.patch
>
>
> Commons-httpclient has long been EOL. Critically, it has several security 
> vulnerabilities: CVE-2012-5783 
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783.
> I saw a recent commit that depends on commons-httpclient for 
> TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency 
> with httpclient APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12710:

Fix Version/s: (was: 2.9.0)
   2.8.0

> Remove dependency on commons-httpclient for TestHttpServerLogs
> --
>
> Key: HADOOP-12710
> URL: https://issues.apache.org/jira/browse/HADOOP-12710
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0
>
> Attachments: HADOOP-12710.001.patch
>
>
> Commons-httpclient has long been EOL. Critically, it has several security 
> vulnerabilities: CVE-2012-5783 
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783.
> I saw a recent commit that depends on commons-httpclient for 
> TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency 
> with httpclient APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380304#comment-15380304
 ] 

Matt Foley commented on HADOOP-12710:
-

Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is not 
yet released.

> Remove dependency on commons-httpclient for TestHttpServerLogs
> --
>
> Key: HADOOP-12710
> URL: https://issues.apache.org/jira/browse/HADOOP-12710
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0
>
> Attachments: HADOOP-12710.001.patch
>
>
> Commons-httpclient has long been EOL. Critically, it has several security 
> vulnerabilities: CVE-2012-5783 
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783.
> I saw a recent commit that depends on commons-httpclient for 
> TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency 
> with httpclient APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects

[Matt Foley (JIRA)]

Matt Foley created HADOOP-13382:
---

 Summary: remove unneeded commons-httpclient dependencies from POM 
files in Hadoop and sub-projects
 Key: HADOOP-13382
 URL: https://issues.apache.org/jira/browse/HADOOP-13382
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Reporter: Matt Foley


The patches for various child and related bugs listed in HADOOP-10105, most 
recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, 
eliminate all use of "commons-httpclient" from Hadoop and its sub-projects 
(except for hadoop-tools/hadoop-openstack; see HADOOP-11614).

However, after incorporating these patches, "commons-httpclient" is still 
listed as a dependency in these POM files:
* hadoop-project/pom.xml
* hadoop-common-project/hadoop-common/pom.xml
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml

We wish to remove these, but since commons-httpclient is still used in many 
files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to
* hadoop-tools/hadoop-openstack/pom.xml
(We'll add a note to HADOOP-11614 to undo this when commons-httpclient is 
removed from hadoop-openstack.)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12665) Document hadoop.security.token.service.use_ip

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088151#comment-15088151
 ] 

Matt Foley commented on HADOOP-12665:
-

In branch-1 it was documented in 
[core-default.xml|https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/core/core-default.xml]
 as:
{code}

  hadoop.security.token.service.use_ip
  true
  Controls whether tokens always use IP addresses.  DNS changes
  will not be detected if this option is enabled.  Existing client connections
  that break will always reconnect to the IP of the original host.  New clients
  will connect to the host's new IP but fail to locate a token.  Disabling
  this option will allow existing and new clients to detect an IP change and
  continue to locate the new host's token.
  

{code}

This resulted in a corresponding entry in 
https://hadoop.apache.org/docs/r1.2.1/core-default.html

Apparently in branch-2 it was removed from core-default.xml, presumably because 
it is a rarely used parameter.  However, it still needs to be documented 
somewhere because it is required for *multi-homed servers* if kerberos security 
is enabled, as seen in certain customer complaints (that have not been reported 
as Apache Jiras since they were resolved as misconfigurations rather than code 
bugs).  I have documented it thus:

bq. Parameters for Security Token service host resolution

bq. In secure multi-homed environments, the following parameter will need to be 
set to false (it is true by default) on both cluster servers and clients (see 
HADOOP-7733), in core-site.xml.  If it is not set correctly, the symptom will 
be inability to submit an application to YARN from an external client (with 
error "client host not a member of the Hadoop cluster"), or even from an 
in-cluster client if server failover occurs.

I'm including this as part of a white paper I'm writing on the whole topic of 
multi-homed support.  I was planning to integrate that into Apache Hadoop docs 
when it is done in a couple weeks.  So [~anu], if you like, you can reassign 
this docs jira to me.

> Document hadoop.security.token.service.use_ip
> -
>
> Key: HADOOP-12665
> URL: https://issues.apache.org/jira/browse/HADOOP-12665
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Anu Engineer
>
> {{hadoop.security.token.service.use_ip}} is not documented in 2.x/trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.006.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.005.patch, HADOOP-12617.006.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617-branch-2.7.002.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.007.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047453#comment-15047453
 ] 

Matt Foley commented on HADOOP-12617:
-

[~vinayrpet], thanks for the review.  Agree re IBM_JAVA, have changed it.  
Rather than mix usages, I also corrected two existing instances of 
'System.getProperty("java.vendor").contains("IBM")', so now KerberosUtil.java 
uses solely the 'IBM_JAVA' usage.  I only did this in the trunk patch, not for 
maintenance branches, since it is a cleanup not a bug fix.

Will run it thru the Hadoop QA robot once more, then I will commit it per your 
+1.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.008.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, 
> HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Status: Open  (was: Patch Available)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, 
> HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-12617.
-
   Resolution: Fixed
Fix Version/s: 2.8.0

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Fix For: 2.8.0
>
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, 
> HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Target Version/s: 2.8.0, 3.0.0  (was: 3.0.0, 2.7.3, 2.9.0)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, 
> HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047768#comment-15047768
 ] 

Matt Foley commented on HADOOP-12617:
-

Happily, the spurious asflicense check went away.  Also, I ran an IBM Java SDK 
(version 8.0) on a Linux CentOS 6 VM, and established that the correct fully 
qualified class name for PrincipalName is 
"com.ibm.security.krb5.PrincipalName", not 
"com.ibm.security.krb5.internal.PrincipalName".  With the corrected path, it 
works as expected.  So I'm uploading one more version of the patchfile, 
HADOOP-12617.008.patch, with that correction, but there's no need to run the 
robot again.

Committing underway.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047854#comment-15047854
 ] 

Matt Foley commented on HADOOP-12617:
-

Committed to trunk as ada9c2c410c15e95,
branch-2 as 472541291bf868c,
branch-2.8 as 05a7fb3a9a02326cb46.

Should therefore be expected in 2.8.0 and all later releases.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, 
> HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617-branch-2.7.003.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.002.patch, HADOOP-12617-branch-2.7.003.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.005.patch, HADOOP-12617.006.patch, 
> HADOOP-12617.007.patch, HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: (was: HADOOP-12617-branch-2.7.002.patch)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, 
> HADOOP-12617.008.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.002.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.001.patch, HADOOP-12617.002.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045271#comment-15045271
 ] 

Matt Foley commented on HADOOP-12617:
-

[~ste...@apache.org], not in a properly configured system.  There is some 
evidence that if run on a system with Kerberos configured on but default realm 
not defined (either via System Property "java.security.krb5.realm" or krb5.conf 
parameter "default_realm"), that getDefaultRealm() may throw an exception.  But 
that's a fairly gross misconfiguration.

Do you think the error should be suppressed without logging?  I had wondered 
why there was no logging in the KerberosUtils class.

Also, does anyone listening have access to the documentation for IBM Java 
package "com.ibm.security.krb5"?  getDomainRealm() follows the model of 
getDefaultRealm(), but I was unable to confirm that 
com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics 
of principal string type "KRB_NT_SRV_HST".  Thanks.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.001.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045271#comment-15045271
 ] 

Matt Foley edited comment on HADOOP-12617 at 12/7/15 5:43 PM:
--

[~ste...@apache.org], not in a properly configured system.  There is some 
evidence that if run on a system with Kerberos configured on but default realm 
not defined (either via System Property "java.security.krb5.realm" or krb5.conf 
parameter "default_realm"), that getDefaultRealm() or getDomainRealm() may 
throw an exception.  But that's a fairly gross misconfiguration.

Do you think the error should be suppressed without logging?  I had wondered 
why there was no logging in the KerberosUtils class.

Also, does anyone listening have access to the documentation for IBM Java 
package "com.ibm.security.krb5"?  getDomainRealm() follows the model of 
getDefaultRealm(), but I was unable to confirm that 
com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics 
of principal string type "KRB_NT_SRV_HST".  Thanks.


was (Author: mattf):
[~ste...@apache.org], not in a properly configured system.  There is some 
evidence that if run on a system with Kerberos configured on but default realm 
not defined (either via System Property "java.security.krb5.realm" or krb5.conf 
parameter "default_realm"), that getDefaultRealm() may throw an exception.  But 
that's a fairly gross misconfiguration.

Do you think the error should be suppressed without logging?  I had wondered 
why there was no logging in the KerberosUtils class.

Also, does anyone listening have access to the documentation for IBM Java 
package "com.ibm.security.krb5"?  getDomainRealm() follows the model of 
getDefaultRealm(), but I was unable to confirm that 
com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics 
of principal string type "KRB_NT_SRV_HST".  Thanks.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.001.patch, HADOOP-12617.002.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.003.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.001.patch, HADOOP-12617.002.patch, HADOOP-12617.003.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045271#comment-15045271
 ] 

Matt Foley edited comment on HADOOP-12617 at 12/7/15 8:32 PM:
--

[~ste...@apache.org], not in a properly configured system.  There is some 
evidence that if run on a system with Kerberos configured on but default realm 
not defined (either via System Property "java.security.krb5.realm" or krb5.conf 
parameter "default_realm"), that getDefaultRealm() or getDomainRealm() may 
throw an exception.  But that's a fairly gross misconfiguration.

Also, I put a call to both getDefaultRealm() and getDomainRealm() in the 
revised unit test, and they did not throw an exception on the Jenkins test 
host, which may or may not be well configured.

Do you think the error should be suppressed without logging?  I had wondered 
why there was no logging in the KerberosUtils class.

Also, does anyone listening have access to the documentation for IBM Java 
package "com.ibm.security.krb5"?  getDomainRealm() follows the model of 
getDefaultRealm(), but I was unable to confirm that 
com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics 
of principal string type "KRB_NT_SRV_HST".  Thanks.


was (Author: mattf):
[~ste...@apache.org], not in a properly configured system.  There is some 
evidence that if run on a system with Kerberos configured on but default realm 
not defined (either via System Property "java.security.krb5.realm" or krb5.conf 
parameter "default_realm"), that getDefaultRealm() or getDomainRealm() may 
throw an exception.  But that's a fairly gross misconfiguration.

Do you think the error should be suppressed without logging?  I had wondered 
why there was no logging in the KerberosUtils class.

Also, does anyone listening have access to the documentation for IBM Java 
package "com.ibm.security.krb5"?  getDomainRealm() follows the model of 
getDefaultRealm(), but I was unable to confirm that 
com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics 
of principal string type "KRB_NT_SRV_HST".  Thanks.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.001.patch, HADOOP-12617.002.patch, HADOOP-12617.003.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: (was: HADOOP-12617.004.patch)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.005.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.005.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.005.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: (was: HADOOP-12617.001.patch)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.004.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: (was: HADOOP-12617.002.patch)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.004.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: (was: HADOOP-12617.004.patch)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.003.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.004.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.001.patch, HADOOP-12617.002.patch, HADOOP-12617.003.patch, 
> HADOOP-12617.004.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.004.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, 
> HADOOP-12617.003.patch, HADOOP-12617.004.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Status: Patch Available  (was: Open)

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.001.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044177#comment-15044177
 ] 

Matt Foley commented on HADOOP-12617:
-

Teradata engineering team provided debugger information leading to the 
following analysis:

A Java client desiring to initiate communication via HTTP/SPNEGO does the 
following:  In the context of a UGI.doAs(), it calls 
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(),
 which in turn calls doSpnegoSequence().

Inside doSpnegoSequence(), is this code at 
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java#L287-L298
 :
{code}
Subject.doAs(subject, new PrivilegedExceptionAction() {
public Void run() throws Exception {
GSSContext gssContext = null;

try {
GSSManager gssManager = GSSManager.getInstance();
String servicePrincipal = KerberosUtil.getServicePrincipal("HTTP", 
KerberosAuthenticator.this.url.getHost());  
Oid oid = KerberosUtil.getOidInstance("NT_GSS_KRB5_PRINCIPAL"); 
GSSName serviceName = gssManager.createName(servicePrincipal, oid);
{code}
Subject is set with the correct user principal and target server name and 
realm.  After executing the line:
{code}
String servicePrincipal = KerberosUtil.getServicePrincipal("HTTP", 
KerberosAuthenticator.this.url.getHost());
 {code}
The ServicePrincipal is returned as: "HTTP/hostname" (with an actual value for 
"hostname"), but no realm.
Then, when it gets the serviceName
{code}
GSSName serviceName = gssManager.createName(servicePrincipal, oid);
{code}
The resulting data structure includes a subfield Krb5PrincipalName with value 
"HTTP/hostname@DEFAULTREALM", where "hostname" and "DEFAULTREALM" are of course 
substituted with real values.  The default realm should not be here.  It is 
required that the correct non-default realm should be derived from the domain 
of "hostname".

However, the second component of a principal is, strictly speaking, the 
"instance", not the "server". Thus, the Kerb libraries are not inferring the 
realm from the domain portion of the server name, because the formal semantics 
don't actually match; and of course there is nowhere else the realm could be 
correctly inferred from.  KerberosUtil.getServicePrincipal() has to return a 
full principal with correct realm, rather than the short principal with 
inferred default realm.


> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Attachment: HADOOP-12617.001.patch
HADOOP-12617-branch-2.7.001.patch

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.001.patch
>
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

Matt Foley created HADOOP-12617:
---

 Summary: SPNEGO authentication request to non-default realm gets 
default realm name inserted in target server principal
 Key: HADOOP-12617
 URL: https://issues.apache.org/jira/browse/HADOOP-12617
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.7.1
 Environment: Java client talking to two secure clusters in different 
Kerberos realms,
or talking to any secure cluster in non-default realm
Reporter: Matt Foley
Assignee: Matt Foley


Note: This is NOT a vulnerability.

In order for a single Java client to communicate with two different secure 
clusters in different realms (only one of which can be the "default_realm"), 
the client's krb5.conf file must specify both realms, and provide a 
\[domain_realm\] section that maps cluster servers' domains to the correct 
realms.  With other appropriate behaviors (such as using the config from each 
cluster to talk to the respective clusters, and a user principal from each 
realm to talk to the respective realms), this is sufficient for most Hadoop 
ecosystem clients.  

But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
talking to a non-default realm.  The default realm name gets incorrectly 
inserted into the construction of the target server principal for the 
non-default-realm cluster.  Details and proposed solution are given in the 
first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044180#comment-15044180
 ] 

Matt Foley commented on HADOOP-12617:
-

The proposed patch provides the required functionality with no changes to API 
signatures or exceptions, and following the model of other code in 
KerberosUtils.java.

Furthermore, I grepped all java files in a broad set of Hadoop ecosystem 
components:
* accumulo, atlas, calcite, datafu, falcon, flume, hadoop, hbase, hive, hue, 
kafka, knox, mahout, oozie, phoenix, pig, ranger, slider, spark, sqoop, storm, 
tez, zookeeper

and found that no code calls KerberosUtils.getServicePrincipal() directly 
except that in the Hadoop auth package.  Both instances there use the result 
ONLY as an argument for an immediate call to gssManager.createName(), which 
accepts the fully qualified principal and does the right thing with it.  So I 
believe this patch is safe.

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12617:

Target Version/s: 3.0.0, 2.7.3, 2.9.0

> SPNEGO authentication request to non-default realm gets default realm name 
> inserted in target server principal
> --
>
> Key: HADOOP-12617
> URL: https://issues.apache.org/jira/browse/HADOOP-12617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.1
> Environment: Java client talking to two secure clusters in different 
> Kerberos realms,
> or talking to any secure cluster in non-default realm
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> Note: This is NOT a vulnerability.
> In order for a single Java client to communicate with two different secure 
> clusters in different realms (only one of which can be the "default_realm"), 
> the client's krb5.conf file must specify both realms, and provide a 
> \[domain_realm\] section that maps cluster servers' domains to the correct 
> realms.  With other appropriate behaviors (such as using the config from each 
> cluster to talk to the respective clusters, and a user principal from each 
> realm to talk to the respective realms), this is sufficient for most Hadoop 
> ecosystem clients.  
> But our SPNEGO using clients, such as Oozie, have a bug when it comes to 
> talking to a non-default realm.  The default realm name gets incorrectly 
> inserted into the construction of the target server principal for the 
> non-default-realm cluster.  Details and proposed solution are given in the 
> first comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-12437) Allow SecurityUtil to lookup alternate hostnames

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-12437:

Environment: multi-homed

> Allow SecurityUtil to lookup alternate hostnames 
> -
>
> Key: HADOOP-12437
> URL: https://issues.apache.org/jira/browse/HADOOP-12437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: net, security
> Environment: multi-homed
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HADOOP-12437.04.patch, HADOOP-12437.05.patch, 
> HDFS-9109.01.patch, HDFS-9109.02.patch, HDFS-9109.03.patch
>
>
> The configuration setting {{dfs.datanode.dns.interface}} lets the DataNode 
> select its hostname by doing a reverse lookup of IP addresses on the specific 
> network interface. This does not work {{when /etc/hosts}} is used to setup 
> alternate hostnames, since {{DNS#reverseDns}} only queries the DNS servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9730) fix hadoop.spec to add task-log4j.properties

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13728989#comment-13728989
 ] 

Matt Foley commented on HADOOP-9730:


Committed to branch-1 on 8/4/2013 while checking for consistency between 1.2.1 
release and branch-1.

 fix hadoop.spec to add task-log4j.properties 
 -

 Key: HADOOP-9730
 URL: https://issues.apache.org/jira/browse/HADOOP-9730
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 1.2.1
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Fix For: 1.2.1

 Attachments: HADOOP-9730.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9730) fix hadoop.spec to add task-log4j.properties

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-9730.


   Resolution: Fixed
Fix Version/s: 1.2.1

+1.  Thanks, Giri, that fixed the problem.

 fix hadoop.spec to add task-log4j.properties 
 -

 Key: HADOOP-9730
 URL: https://issues.apache.org/jira/browse/HADOOP-9730
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 1.2.1
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Fix For: 1.2.1

 Attachments: HADOOP-9730.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9668) Configuration#iterator method should interpret variables in a property as well as Configuration#get method

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-9668:
---

Target Version/s: 2.1.0-beta, 1.3.0  (was: 2.1.0-beta, 1.2.1)

 Configuration#iterator method should interpret variables in a property as 
 well as Configuration#get method
 --

 Key: HADOOP-9668
 URL: https://issues.apache.org/jira/browse/HADOOP-9668
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 2.1.0-beta, 1.2.1
Reporter: Kousuke Saruta
Priority: Minor
  Labels: newbie

 o.a.h.conf.Configuration#get method interpret variables in a property in 
 *-site.xml but Configuration#iterator method doesn't.
 For example, when a property user.name is set to hadoop and another 
 property hadoop.tmp.dir is set to /tmp/${user.name}, Configuration#get 
 interpret hadoop.tmp.dir as /tmp/hadoop but Configuration#iterator 
 doesn't interpret.
 I think Configuration#iterator should interpret variables or if there are 
 some reasons, it should be documented that Configuration#iterator doesn't 
 interpret variables.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9668) Configuration#iterator method should interpret variables in a property as well as Configuration#get method

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708162#comment-13708162
 ] 

Matt Foley commented on HADOOP-9668:


Since work has not yet started, and we are about to produce 1.2.1-rc, deferring 
this to 1.3.0.

 Configuration#iterator method should interpret variables in a property as 
 well as Configuration#get method
 --

 Key: HADOOP-9668
 URL: https://issues.apache.org/jira/browse/HADOOP-9668
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 2.1.0-beta, 1.2.1
Reporter: Kousuke Saruta
Priority: Minor
  Labels: newbie

 o.a.h.conf.Configuration#get method interpret variables in a property in 
 *-site.xml but Configuration#iterator method doesn't.
 For example, when a property user.name is set to hadoop and another 
 property hadoop.tmp.dir is set to /tmp/${user.name}, Configuration#get 
 interpret hadoop.tmp.dir as /tmp/hadoop but Configuration#iterator 
 doesn't interpret.
 I think Configuration#iterator should interpret variables or if there are 
 some reasons, it should be documented that Configuration#iterator doesn't 
 interpret variables.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9504) MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-9504.


  Resolution: Fixed
Target Version/s: 0.23.8, 2.1.0-beta, 1.2.1  (was: 2.1.0-beta, 0.23.8, 
1.2.1)

Committed to branch-1 and branch-1.2.  Thanks, Liang and Jason!

 MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo
 -

 Key: HADOOP-9504
 URL: https://issues.apache.org/jira/browse/HADOOP-9504
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.0.0, 2.0.3-alpha
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Critical
 Fix For: 2.1.0-beta, 1.2.1, 0.23.8

 Attachments: HADOOP-9504-branch-1.txt, HADOOP-9504.txt, 
 HADOOP-9504-v2.txt


 Please see HBASE-8416 for detail information.
 we need to take care of the synchronization for HashMap put(), otherwise it 
 may lead to spin loop.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9504) MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-9504:
---

Fix Version/s: 1.2.1

 MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo
 -

 Key: HADOOP-9504
 URL: https://issues.apache.org/jira/browse/HADOOP-9504
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.0.0, 2.0.3-alpha
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Critical
 Fix For: 2.1.0-beta, 0.23.8, 1.2.1

 Attachments: HADOOP-9504-branch-1.txt, HADOOP-9504.txt, 
 HADOOP-9504-v2.txt


 Please see HBASE-8416 for detail information.
 we need to take care of the synchronization for HashMap put(), otherwise it 
 may lead to spin loop.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666428#comment-13666428
 ] 

Matt Foley commented on HADOOP-9573:


+1, Thanks, Giri!


 Fix test-patch script to work with the enhanced PreCommit-Admin script.
 ---

 Key: HADOOP-9573
 URL: https://issues.apache.org/jira/browse/HADOOP-9573
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 1.0.3
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: 
 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch, 
 hadoop-9573.patch


 test-patch script currently take the latest available patch for a given jira 
 and performs the test. This jira is to enhance test-patch script to take 
 attachment-id of a patch as an input and perform the tests using that 
 attachment-id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662371#comment-13662371
 ] 

Matt Foley commented on HADOOP-9573:


Exchanged comments with Giri:
.bq What happened to CheckStyle?  It looks like it got commented out some time 
ago, but why? Or was it entirely replaced by FindBugs? 

[~gkesavan]: Nope, we never had checkstyle in test-patch working. Right from 
the early days checkstyle function was always commented out,
so this time around I removed it instead of leaving it as a comment.  It would 
be very painful to implement checkstyle at this point in time, 
as making sure hadoop code base is complaint with checkstyle won't be that easy.

 Fix test-patch script to work with the enhanced PreCommit-Admin script.
 ---

 Key: HADOOP-9573
 URL: https://issues.apache.org/jira/browse/HADOOP-9573
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 1.0.3
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: 
 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch


 test-patch script currently take the latest available patch for a given jira 
 and performs the test. This jira is to enhance test-patch script to take 
 attachment-id of a patch as an input and perform the tests using that 
 attachment-id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662390#comment-13662390
 ] 

Matt Foley commented on HADOOP-9573:


Review posted in reviewboard.  Lot of good work, a few things need cleanup.

I assume you've tested this with test Jiras.  Please describe tests done.  
Thanks, Giri.

 Fix test-patch script to work with the enhanced PreCommit-Admin script.
 ---

 Key: HADOOP-9573
 URL: https://issues.apache.org/jira/browse/HADOOP-9573
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 1.0.3
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: 
 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch


 test-patch script currently take the latest available patch for a given jira 
 and performs the test. This jira is to enhance test-patch script to take 
 attachment-id of a patch as an input and perform the tests using that 
 attachment-id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662371#comment-13662371
 ] 

Matt Foley edited comment on HADOOP-9573 at 5/20/13 10:01 PM:
--

Exchanged comments with Giri:
bq. What happened to CheckStyle?  It looks like it got commented out some time 
ago, but why? Or was it entirely replaced by FindBugs? 

[~gkesavan]: Nope, we never had checkstyle in test-patch working. Right from 
the early days checkstyle function was always commented out,
so this time around I removed it instead of leaving it as a comment.  It would 
be very painful to implement checkstyle at this point in time, 
as making sure hadoop code base is complaint with checkstyle won't be that easy.

  was (Author: mattf):
Exchanged comments with Giri:
.bq What happened to CheckStyle?  It looks like it got commented out some time 
ago, but why? Or was it entirely replaced by FindBugs? 

[~gkesavan]: Nope, we never had checkstyle in test-patch working. Right from 
the early days checkstyle function was always commented out,
so this time around I removed it instead of leaving it as a comment.  It would 
be very painful to implement checkstyle at this point in time, 
as making sure hadoop code base is complaint with checkstyle won't be that easy.
  
 Fix test-patch script to work with the enhanced PreCommit-Admin script.
 ---

 Key: HADOOP-9573
 URL: https://issues.apache.org/jira/browse/HADOOP-9573
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 1.0.3
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: 
 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch


 test-patch script currently take the latest available patch for a given jira 
 and performs the test. This jira is to enhance test-patch script to take 
 attachment-id of a patch as an input and perform the tests using that 
 attachment-id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662390#comment-13662390
 ] 

Matt Foley edited comment on HADOOP-9573 at 5/20/13 10:02 PM:
--

Review posted in reviewboard.  Lot of good work, a few things need cleanup.

I assume you've tested this with test Jiras.  Please describe tests done.  
Thanks Giri!

  was (Author: mattf):
Review posted in reviewboard.  Lot of good work, a few things need cleanup.

I assume you've tested this with test Jiras.  Please describe tests done.  
Thanks, Giri.
  
 Fix test-patch script to work with the enhanced PreCommit-Admin script.
 ---

 Key: HADOOP-9573
 URL: https://issues.apache.org/jira/browse/HADOOP-9573
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 1.0.3
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: 
 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch


 test-patch script currently take the latest available patch for a given jira 
 and performs the test. This jira is to enhance test-patch script to take 
 attachment-id of a patch as an input and perform the tests using that 
 attachment-id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8515) Upgrade Jetty to the current Jetty 7 release

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-8515:
---

Target Version/s: 3.0.0, 1.3.0  (was: 1.2.0, 3.0.0)

 Upgrade Jetty to the current Jetty 7 release
 

 Key: HADOOP-8515
 URL: https://issues.apache.org/jira/browse/HADOOP-8515
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Luke Lu
  Labels: jetty
 Attachments: hadoop_jetty.patch.v2


 According to 
 http://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00026.html, jetty-6 
 has been effectively EOL since January. Let's bump jetty to the 7 series. The 
 current jetty 6.1.26 contains at least one known vulnerability: CVE-2011-4461.
 Note this can be an incompatible change if you reference jetty-6 packages 
 (org.mortbay.*).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8515) Upgrade Jetty to the current Jetty 7 release

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656735#comment-13656735
 ] 

Matt Foley commented on HADOOP-8515:


Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 
if you intend to submit a fix for branch-1.2.

 Upgrade Jetty to the current Jetty 7 release
 

 Key: HADOOP-8515
 URL: https://issues.apache.org/jira/browse/HADOOP-8515
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Luke Lu
  Labels: jetty
 Attachments: hadoop_jetty.patch.v2


 According to 
 http://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00026.html, jetty-6 
 has been effectively EOL since January. Let's bump jetty to the 7 series. The 
 current jetty 6.1.26 contains at least one known vulnerability: CVE-2011-4461.
 Note this can be an incompatible change if you reference jetty-6 packages 
 (org.mortbay.*).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8477) Pull in Yahoo! Hadoop Tutorial and update it accordingly.

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-8477:
---

Target Version/s: 3.0.0, 1.3.0  (was: 1.2.0, 3.0.0)

 Pull in Yahoo! Hadoop Tutorial and update it accordingly.
 -

 Key: HADOOP-8477
 URL: https://issues.apache.org/jira/browse/HADOOP-8477
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Affects Versions: 1.1.0, 2.0.0-alpha
Reporter: Robert Joseph Evans
 Attachments: tutorial.tgz


 I was able to get the Yahoo! Hadoop tutorial released under an Apache 2.0 
 license.  This allows us to make it a official part of the Hadoop Project.  
 This ticket is to pull the tutorial and update it as needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8477) Pull in Yahoo! Hadoop Tutorial and update it accordingly.

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656736#comment-13656736
 ] 

Matt Foley commented on HADOOP-8477:


Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 
if you intend to submit a fix for branch-1.2.

 Pull in Yahoo! Hadoop Tutorial and update it accordingly.
 -

 Key: HADOOP-8477
 URL: https://issues.apache.org/jira/browse/HADOOP-8477
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Affects Versions: 1.1.0, 2.0.0-alpha
Reporter: Robert Joseph Evans
 Attachments: tutorial.tgz


 I was able to get the Yahoo! Hadoop tutorial released under an Apache 2.0 
 license.  This allows us to make it a official part of the Hadoop Project.  
 This ticket is to pull the tutorial and update it as needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8161) Support for Azure Storage

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-8161:
---

Target Version/s: 0.24.0, 1.3.0  (was: 1.2.0, 0.24.0)

 Support for Azure Storage
 -

 Key: HADOOP-8161
 URL: https://issues.apache.org/jira/browse/HADOOP-8161
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: native
Reporter: Sanjay Radia



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8161) Support for Azure Storage

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656737#comment-13656737
 ] 

Matt Foley commented on HADOOP-8161:


Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 
if you intend to submit a fix for branch-1.2.

 Support for Azure Storage
 -

 Key: HADOOP-8161
 URL: https://issues.apache.org/jira/browse/HADOOP-8161
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: native
Reporter: Sanjay Radia



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8103) Hadoop-bin commands for windows

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-8103:
---

Target Version/s: 0.24.0, 1.3.0  (was: 1.2.0, 0.24.0)

 Hadoop-bin commands for windows
 ---

 Key: HADOOP-8103
 URL: https://issues.apache.org/jira/browse/HADOOP-8103
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: native
Affects Versions: 1.1.0, 0.24.0
Reporter: Sanjay Radia
 Attachments: windows-cmd-scripts.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8103) Hadoop-bin commands for windows

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656738#comment-13656738
 ] 

Matt Foley commented on HADOOP-8103:


Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 
if you intend to submit a fix for branch-1.2.

 Hadoop-bin commands for windows
 ---

 Key: HADOOP-8103
 URL: https://issues.apache.org/jira/browse/HADOOP-8103
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: native
Affects Versions: 1.1.0, 0.24.0
Reporter: Sanjay Radia
 Attachments: windows-cmd-scripts.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira