[jira] [Commented] (HADOOP-15072) Upgrade Apache Kerby version to 1.1.0
[
https://issues.apache.org/jira/browse/HADOOP-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270376#comment-16270376
]
genericqa commented on HADOOP-15072:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
25m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
9s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15072 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899768/HADOOP-15072-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 1c154313c313 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d331762 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13758/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5000) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13758/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Upgrade Apache Kerby version to 1.1.0
> -
>
> Key: HADOOP-15072
> URL: https://issues.apache.org/jira/browse/HADOOP-15072
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Jiajia Li
>Assignee: Jiajia Li
> Attachments: HADOOP-15072-001.patch
>
>
> Apache Kerby 1.1.0 implements cross-realm support, and also includes a GSSAPI
>
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270412#comment-16270412 ] SammiChen commented on HADOOP-14964: Hi [~chris.douglas], thanks for summarize the discussion here. bq. Please cherry-pick from branch-2.9 to branch-2.8, then from branch-2.8 to branch-2.8.3 so the lineage is clear. Got it. bq. For (1), what are all the transitive dependencies for this module? aliyun-sdk-oss 2.8.1 is used in trunk, branch-2 and branch-2.9. It has three external dependencies, net.sf.json-lib/json-lib(not used in Hadoop), org.apache.httpcomponents/httpclient and org.jdom/jdom(same 1.1 version as Hadoop). For httpclient, Hadoop use 4.5.2, oss sdk use 4.4.1. 4.5.2 fully support the function oss sdk used in 4.4.1. So it's OK for oss-sdk to use the current 4.5.2 httpclient in Hadoop, also httpclient dependency exclusion is declared for oss-sdk in trunk and branch-2.9 to avoid any conflict. I would say basically the oss sdk transitive dependencies is not a problem. bq. For (2), if we take the current branch-2.9 and cut a 2.9.1 release, that would tranquilize anxiety about the upgrade path. SammiChen, would you be able to RM this? Arun Suresh and Subru Krishnan may be able to help by providing pointers to release docs. Thanks for the asking. I would like to take the RM role if possible. Also guide is strongly needed:) bq. As policy, (4) is stickier. Even releasing this with 2.9.1 doesn't strictly adhere to our rules, but it's better than adding another, active release branch. The case for 2.8.3 is more problematic. The cadence for 2.8.x will decelerate more rapidly than 2.9.x, so fixes to Aliyun OSS will be released less often. We may not do its users a favor by including an outdated client with their 2.8 clusters. Frankly, maintenance is also simpler when we disallow feature backports into patch releases, rather than discussing the merits of each one. This can't become a precedent; it takes too much time. I do understand the concerns. Previously we don't know the release process very well, should proposal the request earlier. We will convey community's feedback to Aliyun OSS team and discuss the possible solutions. > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Fix For: 2.9.1 > > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270527#comment-16270527 ] Steve Loughran commented on HADOOP-14475: - bq. let's maybe handle that as a distinct issue from hooking up S3A to metrics2 as a source? your choice. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch, > HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270532#comment-16270532 ] Steve Loughran commented on HADOOP-14475: - bq. Quick glance at code, looks like instrumentation is null. It gets set in initialize() and closed / set to null in close(), so I wonder if this is test code using the FS after close()? Would be easy enough to test. looking at S3A's getFileStatus(), I don't see any check for the FS being open. -the standard entry points do need that check -ContractTestUtils.cleanup() does catch & downgrade all Exceptions to logging, so teardown doesn't hide the real test failure -but looking at ContractTestUtils.rm(), I don't see why it needs to bother with that exists() check at all. It can just do the delete(recursive). > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch, > HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15056: Summary: Fix TestUnbuffer#testUnbufferException failure (was: Fix for TestUnbuffer#testUnbufferException Test) > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15056: Attachment: HADOOP-15056.002.patch Hi [~jackbearden], I think StreamCapabilitiesPolicy#unbuffer should still call hasCapability if the input stream implements interface StreamCapabilities. Uploaded patch 002. Let me know what you think. Testing Done * TestCryptoStreams * TestCryptoStreamsForLocalFS * TestCryptoStreamsNormal * TestHdfsCryptoStreams * TestUnbuffer > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15075) Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.)
madhu raghavendra created HADOOP-15075: -- Summary: Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.) Key: HADOOP-15075 URL: https://issues.apache.org/jira/browse/HADOOP-15075 Project: Hadoop Common Issue Type: New Feature Components: hdfs-client Affects Versions: 3.0.0-alpha3 Reporter: madhu raghavendra Fix For: site Need to implement Knox SSO login feature for hadoop webUIs like HDFS Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have SPNEGO feature enabled, however having Knox SSO login feature seems to be a good feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15073) SequenceFile.Reader will unexpectedly quit next() iterator while the file ends with sync and appended
[ https://issues.apache.org/jira/browse/HADOOP-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Howard Lee updated HADOOP-15073: External issue URL: (was: https://issues.apache.org/jira/browse/HADOOP-7139) > SequenceFile.Reader will unexpectedly quit next() iterator while the file > ends with sync and appended > - > > Key: HADOOP-15073 > URL: https://issues.apache.org/jira/browse/HADOOP-15073 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.6.3, 2.7.4, 2.8.2, 3.0.0-alpha3 >Reporter: Howard Lee > > The SequenceFile.Writer will insert SYNC into file every SYNC_INTERVAL. > In the case that SequenceFile ends with SYNC coincidentally, and another > Writer open it with mode AppendIfExits, there meets the BUG. > For the AppendIfExits set _appendMode _ to _true_ , the init method will > insert another SYNC. In such case, there will be two SYNC MAKR continuously. > In SequenceFile.Reader, the method readRecordLength will only test SYNC once, > when there's two SYNC MARK, the _length_ will be -1(The begining of another > SYNC), which means the end of file causing the next method quit unexpectedly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15073) SequenceFile.Reader will unexpectedly quit next() iterator while the file ends with sync and appended
Howard Lee created HADOOP-15073: --- Summary: SequenceFile.Reader will unexpectedly quit next() iterator while the file ends with sync and appended Key: HADOOP-15073 URL: https://issues.apache.org/jira/browse/HADOOP-15073 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.0.0-alpha3, 2.8.2, 2.7.4, 2.6.3 Reporter: Howard Lee The SequenceFile.Writer will insert SYNC into file every SYNC_INTERVAL. In the case that SequenceFile ends with SYNC coincidentally, and another Writer open it with mode AppendIfExits, there meets the BUG. For the AppendIfExits set _appendMode _ to _true_ , the init method will insert another SYNC. In such case, there will be two SYNC MAKR continuously. In SequenceFile.Reader, the method readRecordLength will only test SYNC once, when there's two SYNC MARK, the _length_ will be -1(The begining of another SYNC), which means the end of file causing the next method quit unexpectedly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15073) SequenceFile.Reader will unexpectedly quit next() iterator while the file ends with sync and appended
[ https://issues.apache.org/jira/browse/HADOOP-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Howard Lee updated HADOOP-15073: External issue URL: https://issues.apache.org/jira/browse/HADOOP-7139 > SequenceFile.Reader will unexpectedly quit next() iterator while the file > ends with sync and appended > - > > Key: HADOOP-15073 > URL: https://issues.apache.org/jira/browse/HADOOP-15073 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.6.3, 2.7.4, 2.8.2, 3.0.0-alpha3 >Reporter: Howard Lee > > The SequenceFile.Writer will insert SYNC into file every SYNC_INTERVAL. > In the case that SequenceFile ends with SYNC coincidentally, and another > Writer open it with mode AppendIfExits, there meets the BUG. > For the AppendIfExits set _appendMode _ to _true_ , the init method will > insert another SYNC. In such case, there will be two SYNC MAKR continuously. > In SequenceFile.Reader, the method readRecordLength will only test SYNC once, > when there's two SYNC MARK, the _length_ will be -1(The begining of another > SYNC), which means the end of file causing the next method quit unexpectedly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15074) SequenceFile#Writer flush does not update the length of the written file.
Mukul Kumar Singh created HADOOP-15074:
--
Summary: SequenceFile#Writer flush does not update the length of
the written file.
Key: HADOOP-15074
URL: https://issues.apache.org/jira/browse/HADOOP-15074
Project: Hadoop Common
Issue Type: Bug
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
SequenceFile#Writer flush does not update the length of the file. This happens
because as part of the flush, {{UPDATE_LENGTH}} flag is not passed to the
DFSOutputStream#hsync.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15075) Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.)
[ https://issues.apache.org/jira/browse/HADOOP-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhu raghavendra updated HADOOP-15075: --- Description: Need to implement Knox SSO login feature for hadoop webUIs like HDFS Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have SPNEGO feature enabled, however having Knox SSO login feature seems to be a good option (was: Need to implement Knox SSO login feature for hadoop webUIs like HDFS Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have SPNEGO feature enabled, however having Knox SSO login feature seems to be a good feature.) > Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.) > -- > > Key: HADOOP-15075 > URL: https://issues.apache.org/jira/browse/HADOOP-15075 > Project: Hadoop Common > Issue Type: New Feature > Components: hdfs-client, security >Affects Versions: 3.0.0-alpha3 >Reporter: madhu raghavendra > Fix For: site > > Original Estimate: 504h > Remaining Estimate: 504h > > Need to implement Knox SSO login feature for hadoop webUIs like HDFS > Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have > SPNEGO feature enabled, however having Knox SSO login feature seems to be a > good option -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15075) Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.)
[ https://issues.apache.org/jira/browse/HADOOP-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay resolved HADOOP-15075. -- Resolution: Not A Problem Closing as not a problem - since JWTRedirectAuthenticationHandler should cover this usecase. See HADOOP-11717. > Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.) > -- > > Key: HADOOP-15075 > URL: https://issues.apache.org/jira/browse/HADOOP-15075 > Project: Hadoop Common > Issue Type: New Feature > Components: hdfs-client, security >Affects Versions: 3.0.0-alpha3 >Reporter: madhu raghavendra > Fix For: site > > Original Estimate: 504h > Remaining Estimate: 504h > > Need to implement Knox SSO login feature for hadoop webUIs like HDFS > Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have > SPNEGO feature enabled, however having Knox SSO login feature seems to be a > good option -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Improvements for Hadoop read from AliyunOSS
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Summary: AliyunOSS: Improvements for Hadoop read from AliyunOSS (was: Improvements for Hadoop read from AliyunOSS) > AliyunOSS: Improvements for Hadoop read from AliyunOSS > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch > > > Currently, read performance is poor when Hadoop reads from AliyunOSS. It > needs about 1min to read 1GB from OSS. > Class AliyunOSSInputStream uses single thread to read data from AliyunOSS, > so we can refactor this by using multi-thread pre read to improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270744#comment-16270744
]
Steve Loughran commented on HADOOP-14556:
-
Daryn, got that patch for this yet?
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
> Attachments: HADOOP-14556-001.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270689#comment-16270689
]
genericqa commented on HADOOP-15056:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 55s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15056 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899785/HADOOP-15056.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9382fdc7c7a8 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270899#comment-16270899
]
Jason Lowe commented on HADOOP-15059:
-
Thanks for taking a look, Daryn!
bq. Not to mention we are now stuck supporting two files. If the format changes
again, do we have to support 3 files?
We would only support each file location as long as that token format is
supported. Just like the alternative "bridge releases" approach, we would
eventually remove support for older token versions (and thus their locations).
We would not be forced to support umpteen files unless we wanted to support
umpteen versions of the token format. The advantage of what I'm calling the
"bridge releases" approach is that there's only one place where tokens are
written, no ugly fallback code, etc., but a significant disadvantage is that it
forces two things:
# a strict upgrade path between releases (i.e.: users must fully update across
all applications to the bridge release before upgrading further).
# the new token format must be introduced long before it can actually be used
We should not underestimate the burden of a bridge release especially if
there's only one magical release. It's not enough to qualify the software
works on the bridge release, one must verify via other means that every part of
the application is using the new bridge release jars. If any part of the
application bundles its own, older Hadoop jars the app will still work fine on
the bridge release but will fail when the cluster upgrades. Therefore it's
difficult for admins and users to know for sure they're ready to move beyond
the bridge release safely because knowing the app runs on the bridge release
isn't sufficient.
The multi-location approach is not pretty at all, but it is much more flexible
in upgrade paths and ability for newer software to leverage the new format once
it's introduced.
bq. The whole point of writing the version into the file is to support multiple
versions.
Yes, and writing that version does enable supporting multiple versions, _but
only for software that understands what those version numbers mean_. Adding a
version number enables newer software to read an older file format, but old
software will not be able to read a newer file format. I only see two options
for supporting older software consuming tokens created by newer software:
# write the new version to a different location and have new software prefer
the other location with fallback
# or force everyone to use the old format until nobody until the old software
is no longer supported and then we switch to the new format in the existing
files.
I see the theoretical problem with a 3.x create->2.x rewrite->3.x read
scenario, although I suspect it would not happen in practice. If it does the
fix is to force the 2.x code to upgrade to 3.x, and that's essentially what
we're requiring with a bridge release.
I'm completely OK with going with the "bridge release" approach for this if we
decide it is the right thing to do here. It could make more sense if indeed
the new token format isn't providing any additional features being leveraged
today that cannot be expressed in the old token format, i.e.: no use-cases are
broken by forcing the old format even in 3.0. There would be no pressing need
to move to the new format, so hopefully this would continue for many 3.x
releases. That could increase the likelihood that everyone has naturally
updated to some version that has v1 support before we start forcing v1 support
on everyone.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
[jira] [Updated] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-15059:
Attachment: HADOOP-15059.004.patch
Thanks for joining the conversation, Allen, and for pointing out the
motivations behind the protobuf change. Do you know of existing use cases that
are relying on the new format?
I completely agree the new format is a great path forward for extensibility and
portability, but unfortunately it breaks a number of existing use cases.
bq. Let's be clear: this is only a problem if one has a bundled
hadoop-common.jar.
It's also important to point out that this is a rather common occurrence.
Besides the typical habit of users running their *-with-dependencies.jar on the
cluster, anyone leveraging the framework-on-HDFS approach will be bitten by
this as soon as the nodemanager upgrades.
Having frameworks deploy via HDFS rather than picking them up from the
nodemanager's jars has proven to be a very useful way to better isolate apps
during cluster rolling upgrades and support multiple versions of the framework
on the cluster simultaneously.
bq. Is the end result of this JIRA going to be that all file formats are locked
forever, regardless of where they come from?
I don't think so. As discussed above, we should be able to remove support for
the Writable format when Hadoop no longer supports 2.x apps. Yes, that's
likely quite a long time, but it does not have to be forever.
bq. Hadoop releases have broken rolling upgrade (and non-rolling upgrades, for
that matter) in the middle of the 2.x stream before by removing things such as
container execution types.
We've completed rolling upgrades across all of our clusters for every minor
release of 2.x since rolling upgrades were first supported in 2.6, so we must
not have hit this landmine. Was this the removal of the dedicated Docker
container executor in favor of the unified Linux executor that does everything?
I'm attaching a patch that implements the "bridge release(s)" approach where
the code supports reading the new format but will write the old format by
default. Code can still request the new format explicitly if necessary. The
main drawback is that we don't get to easily leverage the benefits of the new
format since it's not the default format. However I'm hoping native services
and other things that need the new protobuf format can leverage dtutil to
translate the credentials format for easier consumption.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271067#comment-16271067
]
Allen Wittenauer commented on HADOOP-15059:
---
bq. Yes, and writing that version does enable supporting multiple versions, but
only for software that understands what those version numbers mean.
... and can actually read the file. This makes it more than a cosmetic change.
It's a pretty big deal if one is processing token files with anything other
than Java or both endians. Theoretically, switching from Java serialization to
protobuf should greatly lessen the impact if additions to the token file are
needed. The file format is now something easily portable across a wide variety
of scenarios.
It's probably worth pointing out how well this change goes hand-in-hand with
things such as slider/yarn-native-services. It should enable jobs to directly
interact with the system without needing a 'helper' running next to it.
Let's be clear: this is only a problem if one has a bundled hadoop-common.jar.
Is the end result of this JIRA going to be that all file formats are locked
forever, regardless of where they come from? This seems like a very dark
precedent for the project to take if it ever wants to move forward. [It's
looking more and more like Hadoop will be required to bite the bullet on
log4j2, likely sooner rather than later.]
Also, Hadoop releases have broken rolling upgrade (and non-rolling upgrades,
for that matter) in the middle of the 2.x stream before by removing things such
as container execution types. Where is the actual line? It's clear the
compatibility guidelines aren't really followed/enforced.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HADOOP-15054) upgrade hadoop dependency on commons-codec to 1.11
[ https://issues.apache.org/jira/browse/HADOOP-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271212#comment-16271212 ] Hudson commented on HADOOP-15054: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13289 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13289/]) HADOOP-15054. upgrade hadoop dependency on commons-codec to 1.11. (weichiu: rev 3881c9ef7ef811ca5bac6c090cd8d9a8756f81a4) * (edit) hadoop-project/pom.xml > upgrade hadoop dependency on commons-codec to 1.11 > -- > > Key: HADOOP-15054 > URL: https://issues.apache.org/jira/browse/HADOOP-15054 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-beta1 >Reporter: PJ Fanning >Assignee: Bharat Viswanadham > Fix For: 3.1.0 > > Attachments: HADOOP-15054.00.patch > > > https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-auth/3.0.0-beta1 > retains the dependency on an old commons-codec version (1.4). > And hadoop-common. > Would it be possible to consider an upgrade to 1.11? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14742) Document multi-URI replication Inode for ViewFS
[ https://issues.apache.org/jira/browse/HADOOP-14742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271347#comment-16271347 ] Gera Shegalov commented on HADOOP-14742: Apologies for the delay. Yes, HADOOP-12077 is obviously a starting point. Let us set some deadline :) > Document multi-URI replication Inode for ViewFS > --- > > Key: HADOOP-14742 > URL: https://issues.apache.org/jira/browse/HADOOP-14742 > Project: Hadoop Common > Issue Type: Task > Components: documentation, viewfs >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas > > HADOOP-12077 added client-side "replication" capabilities to ViewFS. Its > semantics and configuration should be documented. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271346#comment-16271346
]
Allen Wittenauer commented on HADOOP-15059:
---
bq. Do you know of existing use cases that are relying on the new format?
Yes, but I can't talk about it without violating NDA. The NM writing both
files would probably be an acceptable middle ground, but I'd need to confer.
bq. However I'm hoping native services and other things that need the new
protobuf format can leverage dtutil to translate the credentials format for
easier consumption.
We should verify dtutil can actually do a translation without talking to a
service:
* old format -> new format
* no krb5 creds required
I suspect append will do it though. A specific 'translate' feature might be
easier on the user.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15054) upgrade hadoop dependency on commons-codec to 1.11
[ https://issues.apache.org/jira/browse/HADOOP-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-15054: - Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Thanks PJ Fanning for proposing the upgrade, and [~bharatviswa] for the patch. Committed the patch to trunk (3.1.0). I've verified that after the change, Hadoop depends on commons-codec 1.11, and I also subjected the patch against our internal test infrastructure to verify it does not break downstream applications (at least for CDH per se). > upgrade hadoop dependency on commons-codec to 1.11 > -- > > Key: HADOOP-15054 > URL: https://issues.apache.org/jira/browse/HADOOP-15054 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-beta1 >Reporter: PJ Fanning >Assignee: Bharat Viswanadham > Fix For: 3.1.0 > > Attachments: HADOOP-15054.00.patch > > > https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-auth/3.0.0-beta1 > retains the dependency on an old commons-codec version (1.4). > And hadoop-common. > Would it be possible to consider an upgrade to 1.11? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15075) Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.)
[ https://issues.apache.org/jira/browse/HADOOP-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhu raghavendra updated HADOOP-15075: --- Component/s: security > Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.) > -- > > Key: HADOOP-15075 > URL: https://issues.apache.org/jira/browse/HADOOP-15075 > Project: Hadoop Common > Issue Type: New Feature > Components: hdfs-client, security >Affects Versions: 3.0.0-alpha3 >Reporter: madhu raghavendra > Fix For: site > > Original Estimate: 504h > Remaining Estimate: 504h > > Need to implement Knox SSO login feature for hadoop webUIs like HDFS > Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have > SPNEGO feature enabled, however having Knox SSO login feature seems to be a > good feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15075) Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.)
[ https://issues.apache.org/jira/browse/HADOOP-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270672#comment-16270672 ] Larry McCay commented on HADOOP-15075: -- [~rmadhu877] - Please take a look at using JWTRedirectAuthenticationHandler. Which is in the hadoop-auth module within hadoop-common-project. HADOOP-11717 introduced this handler specifically for use with KnoxSSO. You will need to enable kerberos on the end HTTP ports but it will use the alternate mechanism of redirecting to a URL from which a JWT based cookie will be returned. I will close this as Won't Fix - feel free to reopen it if you need to. > Implement KnoxSSO for hadoop web UIs (hdfs, yarn, history server etc.) > -- > > Key: HADOOP-15075 > URL: https://issues.apache.org/jira/browse/HADOOP-15075 > Project: Hadoop Common > Issue Type: New Feature > Components: hdfs-client, security >Affects Versions: 3.0.0-alpha3 >Reporter: madhu raghavendra > Fix For: site > > Original Estimate: 504h > Remaining Estimate: 504h > > Need to implement Knox SSO login feature for hadoop webUIs like HDFS > Namenode, Yarn RM, MR2 Job history server, spark etc. I know that we have > SPNEGO feature enabled, however having Knox SSO login feature seems to be a > good option -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270747#comment-16270747
]
Steve Loughran commented on HADOOP-14556:
-
Oh, as this doesn't work with session tokens as the current auth mech, it'd be
nice just to pick up pass on those session credentials as is, adding in any
encryption secrets. That way, if I am logged in with some session credentials,
I can still pass those down to submitted jobs
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
> Attachments: HADOOP-14556-001.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15042) Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0
[
https://issues.apache.org/jira/browse/HADOOP-15042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15042:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.10.0
3.0.0
Status: Resolved (was: Patch Available)
Resolved since this seems to be committed.
> Azure PageBlobInputStream.skip() can return negative value when
> numberOfPagesRemaining is 0
> ---
>
> Key: HADOOP-15042
> URL: https://issues.apache.org/jira/browse/HADOOP-15042
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 3.0.0, 2.10.0
>
> Attachments: HADOOP-15042.001.patch
>
>
> {{PageBlobInputStream::skip-->skipImpl}} returns negative values when
> {{numberOfPagesRemaining=0}}. This can cause wrong position to be set in
> NativeAzureFileSystem::seek() and can lead to errors.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271676#comment-16271676 ] Jack Bearden commented on HADOOP-15056: --- Forgive me if I am being too pedantic... The way I see it, by guarding with the hasCapability for StreamCapabilities.UNBUFFER, we may effectively be neglecting the possibility of ever encountering that UnsupportedOperationException. That may result in the user no longer getting the error message in their logs. Are there any other use cases where that exception would be thrown? Note: it appears you fixed this with the latest patch but I am still trying to understand why. Thank you for your patience > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271668#comment-16271668
]
Aaron Fabbri commented on HADOOP-14475:
---
I added some instrumentation to detect use after close and it is not
reproducing now. I also deleted my maven repo to make sure I had a clean
build, though.
Now I do see an assertion failure when running {{mvn clean verify -Dtest=none
-Ds3guard -Dparallel-tests}}
{noformat}
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.379 s
<<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AMetrics
[ERROR] testMetricsRegister(org.apache.hadoop.fs.s3a.ITestS3AMetrics) Time
elapsed: 1.379 s <<< FAILURE!
java.lang.AssertionError: No metrics under test fs for S3AMetrics1-fabbri-new
at
org.apache.hadoop.fs.s3a.ITestS3AMetrics.testMetricsRegister(ITestS3AMetrics.java:41)
{noformat}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch,
> HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch,
> HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271677#comment-16271677 ] Aaron Fabbri commented on HADOOP-13974: --- This patch probably needs significant rewrite now that HADOOP-13786 is merged. I'll work on it soon. > S3a CLI to support list/purge of pending multipart commits > -- > > Key: HADOOP-13974 > URL: https://issues.apache.org/jira/browse/HADOOP-13974 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Aaron Fabbri > Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch, > HADOOP-13974.003.patch > > > The S3A CLI will need to be able to list and delete pending multipart > commits. > We can do the cleanup already via fs.s3a properties. The CLI will let scripts > stat for outstanding data (have a different exit code) and permit batch jobs > to explicitly trigger cleanups. > This will become critical with the multipart committer, as there's a > significantly higher likelihood of commits remaining outstanding. > We may also want to be able to enumerate/cancel all pending commits in the FS > tree -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14444) New implementation of ftp and sftp filesystems
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Waldmann updated HADOOP-1: Status: In Progress (was: Patch Available) > New implementation of ftp and sftp filesystems > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: Lukas Waldmann >Assignee: Lukas Waldmann > Attachments: HADOOP-1.10.patch, HADOOP-1.2.patch, > HADOOP-1.3.patch, HADOOP-1.4.patch, HADOOP-1.5.patch, > HADOOP-1.6.patch, HADOOP-1.7.patch, HADOOP-1.8.patch, > HADOOP-1.9.patch, HADOOP-1.patch > > > Current implementation of FTP and SFTP filesystems have severe limitations > and performance issues when dealing with high number of files. Mine patch > solve those issues and integrate both filesystems such a way that most of the > core functionality is common for both and therefore simplifying the > maintainability. > The core features: > * Support for HTTP/SOCKS proxies > * Support for passive FTP > * Support for explicit FTPS (SSL/TLS) > * Support of connection pooling - new connection is not created for every > single command but reused from the pool. > For huge number of files it shows order of magnitude performance improvement > over not pooled connections. > * Caching of directory trees. For ftp you always need to list whole directory > whenever you ask information about particular file. > Again for huge number of files it shows order of magnitude performance > improvement over not cached connections. > * Support of keep alive (NOOP) messages to avoid connection drops > * Support for Unix style or regexp wildcard glob - useful for listing a > particular files across whole directory tree > * Support for reestablishing broken ftp data transfers - can happen > surprisingly often -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271601#comment-16271601
]
John Zhuge commented on HADOOP-15056:
-
StreamCapabilities was added by HDFS-11644. It is considered a more flexible
approach than {{instanceof}}.
bq. it may prevent helpful debugging information to be passed back to the user
Could you elaborate?
> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for
> the very first time. :)
> I pulled down Hadoop today and when running the tests I encountered a failure
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this
> test case may have been overlooked. Using today's git commit
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case,
> there is an expect mock for an exception UnsupportedOperationException that
> is no longer being thrown.
> It would appear that a test like this would be valuable so my initial
> proposed patch did not remove it. Instead, I removed the conditions that were
> guarding the cast from being able to fire -- as was the previous behavior.
> Now when we encounter an object that doesn't have the UNBUFFERED
> StreamCapability, it will throw an error as it did prior to the recent
> changes.
> Please review and let me know what you think! :D
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271638#comment-16271638 ] Andrew Wang commented on HADOOP-15058: -- [~aw] do you mind reviewing this? > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14444) New implementation of ftp and sftp filesystems
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Waldmann updated HADOOP-1: Attachment: HADOOP-1.11.patch > New implementation of ftp and sftp filesystems > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: Lukas Waldmann >Assignee: Lukas Waldmann > Attachments: HADOOP-1.10.patch, HADOOP-1.11.patch, > HADOOP-1.2.patch, HADOOP-1.3.patch, HADOOP-1.4.patch, > HADOOP-1.5.patch, HADOOP-1.6.patch, HADOOP-1.7.patch, > HADOOP-1.8.patch, HADOOP-1.9.patch, HADOOP-1.patch > > > Current implementation of FTP and SFTP filesystems have severe limitations > and performance issues when dealing with high number of files. Mine patch > solve those issues and integrate both filesystems such a way that most of the > core functionality is common for both and therefore simplifying the > maintainability. > The core features: > * Support for HTTP/SOCKS proxies > * Support for passive FTP > * Support for explicit FTPS (SSL/TLS) > * Support of connection pooling - new connection is not created for every > single command but reused from the pool. > For huge number of files it shows order of magnitude performance improvement > over not pooled connections. > * Caching of directory trees. For ftp you always need to list whole directory > whenever you ask information about particular file. > Again for huge number of files it shows order of magnitude performance > improvement over not cached connections. > * Support of keep alive (NOOP) messages to avoid connection drops > * Support for Unix style or regexp wildcard glob - useful for listing a > particular files across whole directory tree > * Support for reestablishing broken ftp data transfers - can happen > surprisingly often -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271579#comment-16271579
]
genericqa commented on HADOOP-15059:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 2s{color} | {color:orange} root: The patch generated 1 new + 20 unchanged -
4 fixed = 21 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}218m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.fs.TestUnbuffer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15059 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899852/HADOOP-15059.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b102d346e302 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272082#comment-16272082
]
Dapeng Sun commented on HADOOP-10768:
-
HmacSHA1:
{noformat}
Total calls per second: 49716.0
CPU time per call on client: 19397 ns
CPU time per call on server: 37601 ns
{noformat}
HmacMD5:
{noformat}
Total calls per second: 50543.0
CPU time per call on client: 19293 ns
CPU time per call on server: 36630 ns
{noformat}
Configuration of RPCCallBenchmark:
{noformat}
"--clientThreads", "30",
"--serverThreads", "30",
"--warmup", "60",
"--time", "60",
"--serverReaderThreads", "4",
"--messageSize", "1024",
"--qop", "INTEGRITY",
"--cipher","AES/CTR/NoPadding",
"--engine", "protobuf",
"--sasl"
{noformat}
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272093#comment-16272093
]
Vinod Kumar Vavilapalli commented on HADOOP-15059:
--
Tx for taking this up [~jlowe]!
bq. I'm attaching a patch that implements the "bridge release(s)" approach
where the code supports reading the new format but will write the old format by
default.
+1 for the bridging approach.
bq. The main drawback is that we don't get to easily leverage the benefits of
the new format since it's not the default format.
I realized it's just not about changing the default format.
ContainerLaunchContext.tokens in YARN is unfortunately a byte-buffer. Taking a
protobuf, wrapping it into a byte-buffer and sending it to the RM is backwards
to me. The right way to use this in YARN is to assume that the existing tokens
field is old-style credentials and then add a new first-class protobuf based
Credentials field.
The patch looks mostly good to me.
bq. minor suggestion of the private statics for version can be replaced with
the enums
If we are no longer going to bump up this version in the protobuf world, +1 -
there will be only two of these ever. IAC, these are private.
One other minor suggestion if you are doing the above.
Credentials.SerializedFormat can be static.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13493) Compatibility Docs should clarify the policy for what takes precedence when a conflict is found
[
https://issues.apache.org/jira/browse/HADOOP-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271942#comment-16271942
]
Andrew Wang commented on HADOOP-13493:
--
Hey folks, is this still planned for 3.0.0? The blockers are making nice
progress, so would like to close on this too.
> Compatibility Docs should clarify the policy for what takes precedence when a
> conflict is found
> ---
>
> Key: HADOOP-13493
> URL: https://issues.apache.org/jira/browse/HADOOP-13493
> Project: Hadoop Common
> Issue Type: Task
> Components: documentation
>Affects Versions: 2.7.2
>Reporter: Robert Kanter
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: HADOOP-13493.001.patch, HADOOP-13493.002.patch
>
>
> The Compatibility Docs
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Compatibility.html#Java_API)
> list the policies for Private, Public, not annotated, etc Classes and
> members, but it doesn't say what happens when there's a conflict. We should
> try obviously try to avoid this situation, but it would be good to explicitly
> state what takes precedence.
> As an example, until YARN-3225 made it consistent, {{RefreshNodesRequest}}
> looked like this:
> {code:java}
> @Private
> @Stable
> public abstract class RefreshNodesRequest {
> @Public
> @Stable
> public static RefreshNodesRequest newInstance() {
> RefreshNodesRequest request =
> Records.newRecord(RefreshNodesRequest.class);
> return request;
> }
> }
> {code}
> Note that the class is marked {{\@Private}}, but the method is marked
> {{\@Public}}.
> In this example, I'd say that the class level should have priority.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272105#comment-16272105
]
Allen Wittenauer commented on HADOOP-15059:
---
bq. I can't tell my customers "I'm sorry, we're taking a hard downtime because
changed. There's no value add for you".
If only YARN had a system whereby one could dynamically label a node based upon
the current software stack. Then one could schedule around that and/or set up
distributed cache content to match. Very similar to how PBS-based such as
torque have had for 20+ years ...
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272132#comment-16272132
]
Genmao Yu commented on HADOOP-15039:
[~drankye] Could you please keep an eye on it. IMHO, this patch is ready to
merge, and HADOOP-14999 and HADOOP-15027 is pending on it.
> move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272135#comment-16272135
]
Jack Bearden commented on HADOOP-15056:
---
Ah ok, I understand. What are your thoughts on something like the following?
{code:java}
public static void unbuffer(InputStream in) {
try {
((CanUnbuffer) in).unbuffer();
} catch (ClassCastException e) {
LOG.warn("claims to unbuffer but forgets to implement CanUnbuffer");
}
}
{code}
That way, the user would be able to turn on this verbosity if they want it
without it always being squelched. There is one problem however... I noticed
that InputStream doesn't have a logger object. I'll keep looking. Let me know
what you think
> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for
> the very first time. :)
> I pulled down Hadoop today and when running the tests I encountered a failure
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this
> test case may have been overlooked. Using today's git commit
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case,
> there is an expect mock for an exception UnsupportedOperationException that
> is no longer being thrown.
> It would appear that a test like this would be valuable so my initial
> proposed patch did not remove it. Instead, I removed the conditions that were
> guarding the cast from being able to fire -- as was the previous behavior.
> Now when we encounter an object that doesn't have the UNBUFFERED
> StreamCapability, it will throw an error as it did prior to the recent
> changes.
> Please review and let me know what you think! :D
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15073) SequenceFile.Reader will unexpectedly quit next() iterator while the file ends with sync and appended
[ https://issues.apache.org/jira/browse/HADOOP-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272090#comment-16272090 ] Howard Lee commented on HADOOP-15073: - There could be several SYNC MARK continuously, when you open a SequenceFile in append mode, but write nothing before close. > SequenceFile.Reader will unexpectedly quit next() iterator while the file > ends with sync and appended > - > > Key: HADOOP-15073 > URL: https://issues.apache.org/jira/browse/HADOOP-15073 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.6.3, 2.7.4, 2.8.2, 3.0.0-alpha3 >Reporter: Howard Lee > > The SequenceFile.Writer will insert SYNC into file every SYNC_INTERVAL. > In the case that SequenceFile ends with SYNC coincidentally, and another > Writer open it with mode AppendIfExits, there meets the BUG. > For the AppendIfExits set _appendMode _ to _true_ , the init method will > insert another SYNC. In such case, there will be two SYNC MAKR continuously. > In SequenceFile.Reader, the method readRecordLength will only test SYNC once, > when there's two SYNC MARK, the _length_ will be -1(The begining of another > SYNC), which means the end of file causing the next method quit unexpectedly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272192#comment-16272192 ] John Zhuge commented on HADOOP-15056: - IMHO, it is not a good idea to swallow CCE and just log. In the try block, StreamCapabilities#hasCapabilities has to be called, because this is the new algorithm introduced by HADOOP-15012.: an input stream should be queried about its capability UNBUFFER before unbuffer is called. Casting to CanUnbuffer should always succeed in this case. > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272219#comment-16272219
]
Jack Bearden commented on HADOOP-15056:
---
What about the streams that call unbuffer() without the UNBUFFER cap? If the
CanUnbuffer cast always succeeds due to the query, there is no reporting
happening to inform the user that there is a problem. The user would not know
that unbuffer() isn't working on their poorly implemented stream and ultimately
miss out on the resource benefits and never know about it. You see what I mean?
How about something like this?
{code:java}
public static void unbuffer(InputStream in) {
try {
if (in instanceof StreamCapabilities
&& ((StreamCapabilities)
in).hasCapability(StreamCapabilities.UNBUFFER)) {
((CanUnbuffer) in).unbuffer();
} else {
LOG.warn("claims to unbuffer but forgets to implement CanUnbuffer");
}
} catch (ClassCastException e) {
throw new UnsupportedOperationException("failed to cast input stream to
unbuffer, "
+ "you should never see this");
}
}
{code}
> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for
> the very first time. :)
> I pulled down Hadoop today and when running the tests I encountered a failure
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this
> test case may have been overlooked. Using today's git commit
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case,
> there is an expect mock for an exception UnsupportedOperationException that
> is no longer being thrown.
> It would appear that a test like this would be valuable so my initial
> proposed patch did not remove it. Instead, I removed the conditions that were
> guarding the cast from being able to fire -- as was the previous behavior.
> Now when we encounter an object that doesn't have the UNBUFFERED
> StreamCapability, it will throw an error as it did prior to the recent
> changes.
> Please review and let me know what you think! :D
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Bearden updated HADOOP-15056: -- Attachment: HADOOP-15056.003.patch > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, > HADOOP-15056.003.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272261#comment-16272261 ] Jack Bearden commented on HADOOP-15056: --- Submitted patch #3 based on our latest comments. Let me know what you think > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, > HADOOP-15056.003.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14444) New implementation of ftp and sftp filesystems
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Waldmann updated HADOOP-1: Status: Patch Available (was: In Progress) Rebase on top of latest trunk. Removal of clientkeystore - create keystore on the fly > New implementation of ftp and sftp filesystems > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: Lukas Waldmann >Assignee: Lukas Waldmann > Attachments: HADOOP-1.10.patch, HADOOP-1.11.patch, > HADOOP-1.2.patch, HADOOP-1.3.patch, HADOOP-1.4.patch, > HADOOP-1.5.patch, HADOOP-1.6.patch, HADOOP-1.7.patch, > HADOOP-1.8.patch, HADOOP-1.9.patch, HADOOP-1.patch > > > Current implementation of FTP and SFTP filesystems have severe limitations > and performance issues when dealing with high number of files. Mine patch > solve those issues and integrate both filesystems such a way that most of the > core functionality is common for both and therefore simplifying the > maintainability. > The core features: > * Support for HTTP/SOCKS proxies > * Support for passive FTP > * Support for explicit FTPS (SSL/TLS) > * Support of connection pooling - new connection is not created for every > single command but reused from the pool. > For huge number of files it shows order of magnitude performance improvement > over not pooled connections. > * Caching of directory trees. For ftp you always need to list whole directory > whenever you ask information about particular file. > Again for huge number of files it shows order of magnitude performance > improvement over not cached connections. > * Support of keep alive (NOOP) messages to avoid connection drops > * Support for Unix style or regexp wildcard glob - useful for listing a > particular files across whole directory tree > * Support for reestablishing broken ftp data transfers - can happen > surprisingly often -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271842#comment-16271842
]
Daryn Sharp commented on HADOOP-15059:
--
I'm +1 on Jason's patch – with the minor suggestion of the private statics for
version can be replaced with the enums. Anyone object?
I think my statements earlier may have been misconstrued. I'm not proposing a
"dark precedent" unless it's going to be marvel vs a skynet powered by hadoop
movie. All kidding aside...
I'm perfectly fine with the format -being- becoming a PB. Allow existing large
scale deployments a rolling upgrade path or 3.x is DOA. I can't tell my
customers "I'm sorry, we're taking a hard downtime because
changed. There's no value add for you". Let's
gracefully phase it in.
If some limited use cases absolutely positively need it, beyond what the
cmdline can do, then either the impetus is on them, or let's find a way to
force the new output format before it becomes the default. Those that don't
care about incompatibility can accept the risk, while allowing the rest of the
community a stable transition.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14444) New implementation of ftp and sftp filesystems
[
https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272071#comment-16272071
]
genericqa commented on HADOOP-1:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 19s{color}
| {color:red} root generated 3 new + 1237 unchanged - 0 fixed = 1240 total (was
1237) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 45s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 5m
2s{color} | {color:red} root generated 10 new + 4223 unchanged - 0 fixed = 4233
total (was 4223) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 29s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}238m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-1 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899891/HADOOP-1.11.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux 86cfffb94544 3.13.0-135-generic #184-Ubuntu SMP
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272053#comment-16272053
]
Dapeng Sun commented on HADOOP-10768:
-
Hi [~daryn] and [~atm], here is the hash
[performance|https://en.wikipedia.org/wiki/SHA-3#Comparison_of_SHA_functions],
it is hard to improve the performance of MD5, I think we should keep the
solution in current patch. Do you have any suggestions or comments?
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272100#comment-16272100 ] John Zhuge commented on HADOOP-15056: - No worry. Good discussion. IMPALA-5909 complained excessive UOE exceptions. > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271432#comment-16271432 ] Jack Bearden commented on HADOOP-15056: --- Hey [~jzhuge], hope your vacation was great. Thanks for taking a look! I applied the patch and the tests worked for me too. Prior to the inverting the instanceof call I noticed we were creating a situation where the exception would not be fired. I trust your judgement on the hasCapability guard, and maybe this is me just being naive, but I am not sure I see the value. It appears to me as it may prevent helpful debugging information to be passed back to the user. What am I missing? Was it just being too noisy? > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271494#comment-16271494
]
Jason Lowe commented on HADOOP-15059:
-
bq. We should verify dtutil can actually do a translation without talking to a
service:
I verified that I could fetch an HDFS delegation token from one of our secure
clusters via {{hdfs fetchdt}}, copy the old format token file to a machine
that's not part of the Kerberos realm (and not kinit'd at all), and get
{{dtutil append -format protobuf}} to translate the file.
bq. A specific 'translate' feature might be easier on the user.
Is that something I should do as part of this JIRA, or is the latest patch
approach not going to work for other reasons?
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14742) Document multi-URI replication Inode for ViewFS
[ https://issues.apache.org/jira/browse/HADOOP-14742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271339#comment-16271339 ] Zhe Zhang commented on HADOOP-14742: Looks like the Description of HADOOP-12077 is a good starting point? > Document multi-URI replication Inode for ViewFS > --- > > Key: HADOOP-14742 > URL: https://issues.apache.org/jira/browse/HADOOP-14742 > Project: Hadoop Common > Issue Type: Task > Components: documentation, viewfs >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas > > HADOOP-12077 added client-side "replication" capabilities to ViewFS. Its > semantics and configuration should be documented. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271327#comment-16271327
]
Ray Chiang commented on HADOOP-15059:
-
[~daryn]'s suggestion for 3.0->3.1 bridging isn't a perfect solution, but given
the current status (HADOOP-12563 already checked in, this current issue
blocking 3.0.0 GA, not wanting to hold up 3.0.0 for any sort of major
redesign), I don't see a better compromise than to deviate from the
compatibility guidelines for this issue.
Had I caught the issue much earlier during all the protobuf/JACC issue reviews,
there could have been several possibilities for more rolling upgrade friendly
changes to the DT file.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch, HADOOP-15059.004.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
