[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363904#comment-16363904 ] genericqa commented on HADOOP-15176: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 11 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 27s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 14s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 14s{color} | {color:red} root generated 2 new + 1232 unchanged - 2 fixed = 1234 total (was 1234) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 9s{color} | {color:orange} root: The patch generated 8 new + 16 unchanged - 0 fixed = 24 total (was 16) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 22s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 26s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 47s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 97m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.http.TestHttpServerWithSpengo | | | hadoop.security.token.delegation.web.TestWebDelegationToken | | | hadoop.log.TestLogLevel | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-15176 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910566/HADOOP-15176-004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HADOOP-15223: - Status: Patch Available (was: In Progress) > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363890#comment-16363890 ] fang zhenyi edited comment on HADOOP-15223 at 2/14/18 12:58 PM: Thanks [~ajisakaa] for comments.I have remove all of {{Collections.EMPTY_MAP}},{{Collections.EMPTY_SET}} , {{Collections.emptyList()}} and unnecessary {{@SuppressWarning("unchecked")}} annotations.Hope you can review again, thanks a lot. was (Author: zhenyi): Thanks [~ajisakaa] for comments.I have remove all of {{Collections.EMPTY_MAP}},{{Collections.EMPTY_SET}} , {{Collections.emptyList()}} and unnecessary @SuppressWarning("unchecked") annotations.Hope you can review again, thanks a lot. > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[ https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363746#comment-16363746 ] Steve Loughran commented on HADOOP-10571: - OK, submitted the new patch. what do we do about the two checkstyles here...they are indentation.Are they on the new lines though, or is it just that the entire block is indented wrong? If the latter, we don't need to worry. > Use Log.*(Object, Throwable) overload to log exceptions > --- > > Key: HADOOP-10571 > URL: https://issues.apache.org/jira/browse/HADOOP-10571 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Andras Bokor >Priority: Major > Attachments: HADOOP-10571.01.patch, HADOOP-10571.01.patch, > HADOOP-10571.02.patch, HADOOP-10571.03.patch, HADOOP-10571.04.patch, > HADOOP-10571.05.patch, HADOOP-10571.06.patch, HADOOP-10571.07.patch > > > When logging an exception, we often convert the exception to string or call > {{.getMessage}}. Instead we can use the log method overloads which take > {{Throwable}} as a parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14832) Listing s3a bucket without credentials gives Interrupted error
[ https://issues.apache.org/jira/browse/HADOOP-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363757#comment-16363757 ] Steve Loughran commented on HADOOP-14832: - I don't see that, not with my diagnostics entry point. This stack is something we need in the troubleshooting docs though {code} org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on hwdev-steve-ireland-new: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:174) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:365) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:301) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) at org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:256) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:197) at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:333) at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:343) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:308) at org.apache.hadoop.util.RunJar.main(RunJar.java:222) Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:139) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1163) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4990) at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4964) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4213) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150) at
[jira] [Updated] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15176: Status: Patch Available (was: Open) patch 004. Fix Test failure by moving test case to ITestAssumeRole; commit all the checkstyle fixes this time Tested twice * without any credentials: Test* completes, ITest* skipped * with credentials, s3 ireland > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363859#comment-16363859 ] ASF GitHub Bot commented on HADOOP-13972: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/339 apart from the comments inline, I've got nothing else to add...lets see what yetus says. Good to see from the cloudstore test that the option provenance is being set, as it will help debugging in a world where things get more complex > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15231) WavefrontSink for Hadoop Metrics2
[ https://issues.apache.org/jira/browse/HADOOP-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15231: Status: Patch Available (was: Open) Ignoring details like code style & state & lack of tests, why do you need to add the sink into hadoop, rather than just as an adjacent JAR in its own package? That should work, and will avoid the problems you'll encounter about persuading any backports. FWIW, I can see the merits in HADOOP-15230, but here, it's adding something essentially untestable, which the hadoop project would, by adding the class into our codebase, we'd be taking on the responsibility of maintaining. I really doubt anyone will be willing to do that > WavefrontSink for Hadoop Metrics2 > - > > Key: HADOOP-15231 > URL: https://issues.apache.org/jira/browse/HADOOP-15231 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Howard Yoo >Priority: Minor > Labels: metrics, wavefront > > Wavefront is a SaaS based large scale real time metrics monitoring and > analytic system capable of monitoring many different source systems. There > are several Sinks available in Hadoop in order to capture various metrics to > external system. The Wavefront data format follows a similar format as that > of Graphite, with the addition of native point tag support and source value. > The details are outlined here: > [https://docs.wavefront.com/wavefront_data_format.html] > It would be greatly helpful for both Hadoop and Wavefront to have a native > integration using Wavefront Sink to send Hadoop related metrics to Wavefront > Server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363857#comment-16363857 ] ASF GitHub Bot commented on HADOOP-13972: - Github user steveloughran commented on a diff in the pull request: https://github.com/apache/hadoop/pull/339#discussion_r168154261 --- Diff: hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java --- @@ -68,12 +69,16 @@ import static org.apache.hadoop.fs.adl.AdlConfKeys.*; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; --- End diff -- statics at the bottom; these should go into the same block as com.*, ideally in order. > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HADOOP-15223: - Status: In Progress (was: Patch Available) > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HADOOP-15223: - Attachment: HADOOP-15223.002.patch > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363945#comment-16363945 ] genericqa commented on HADOOP-15223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 5m 28s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-15223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910576/HADOOP-15223.002.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14117/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363801#comment-16363801 ] genericqa commented on HADOOP-12502: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 52s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 23s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} hadoop-common-project/hadoop-common: The patch generated 0 new + 289 unchanged - 1 fixed = 289 total (was 290) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 32s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 88m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.http.TestHttpServerWithSpengo | | | hadoop.security.token.delegation.web.TestWebDelegationToken | | | hadoop.log.TestLogLevel | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-12502 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910547/HADOOP-12502-10.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 03f307fcb396 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 60971b8 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/14115/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14115/testReport/ | | Max. process+thread count | 1411 (vs. ulimit of 5500) | | modules | C:
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[ https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363830#comment-16363830 ] Steve Loughran commented on HADOOP-15230: - you need to hit the submit patch button to have Yetus do a build; things won't get reviewed until its happy about all the redline items, and the orangeline warnings are either addressed or justified > org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly > > > Key: HADOOP-15230 > URL: https://issues.apache.org/jira/browse/HADOOP-15230 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Reporter: Howard Yoo >Priority: Major > Labels: GraphiteSink,, metrics > > org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems > that would make it to generate metrics incorrectly. > The problem lies with line 77 ~ 84 of the GraphiteSink java: > {code:java} > for (MetricsTag tag : record.tags()) { > if (tag.value() != null) { > metricsPathPrefix.append("."); > metricsPathPrefix.append(tag.name()); > metricsPathPrefix.append("="); > metricsPathPrefix.append(tag.value()); > } > } > {code} > It produces point tags having name=value pair in the metrics. However, notice > how the tags are added with '.' as its delimiters. Rather than using the '.' > character, it should follow the following convention mentioned in the latest > graphite doc of using ';' character. > [http://graphite.readthedocs.io/en/latest/tags.html] > Also, the value is not properly being escaped, meaning that if the value has > a '.' character in it, it will easily confuse Graphite to accept it as a > delimiter, rather than the value. A really good prime example is when the > value is a hostname or ip address, > {code:java} > metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code} > In this example, the since the value of the hostname contains '.', it is > extremely hard for the receiving end to determine which part is hostname and > which part is the rest of the metrics name. A good strategy is to convert any > '.' character in the value to be converted to other characters, such as '_'. > However, the best way would be to follow the latest metrics convention of > using ';' > {code:java} > metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363854#comment-16363854 ] ASF GitHub Bot commented on HADOOP-13972: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/339 OK. Go the Hadoop JIRA, hit "submit patch" and see what it says. I Can see it'll be complaining about the javadocs, but let's see what else. > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14832) Listing s3a bucket without credentials gives Interrupted error
[ https://issues.apache.org/jira/browse/HADOOP-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363793#comment-16363793 ] Steve Loughran commented on HADOOP-14832: - {code} > bin/hadoop fs -ls s3a://landsat-pds/ ... many lines excluded .. ls: doesBucketExist on landsat-pds: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint {code} The full stack shows lots of retries before things give up, which could be reduced by recognition that there is no point retrying. Filed HADOOP-15232 for it. Tested with the network pulled out, to see if that causes the error you see. No {code} Caused by: java.net.ConnectException: Network is unreachable (connect failed) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) {code} John, I'm going to close as cannot reproduce. If you can try again and do see it, maybe its related to some network timeouts on the retries, so that the retry loop was taking so long that the op was failing. > Listing s3a bucket without credentials gives Interrupted error > -- > > Key: HADOOP-14832 > URL: https://issues.apache.org/jira/browse/HADOOP-14832 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: John Zhuge >Priority: Minor > > In trunk pseudo distributed mode, without setting s3a credentials, listing an > s3a bucket only gives "Interrupted" error : > {noformat} > $ hadoop fs -ls s3a://bucket/ > ls: Interrupted > {noformat} > In comparison, branch-2 gives a much better error message: > {noformat} > (branch-2)$ hadoop_env hadoop fs -ls s3a://bucket/ > ls: doesBucketExist on hdfs-cce: com.amazonaws.AmazonClientException: No AWS > Credentials provided by BasicAWSCredentialsProvider > EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : > com.amazonaws.SdkClientException: Unable to load credentials from service > endpoint > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15176: Attachment: HADOOP-15176-004.patch > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363852#comment-16363852 ] Steve Loughran commented on HADOOP-13761: - I'll call that a success. Is the patch ready to go in? > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761.001.patch, HADOOP-13761.002.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-14451: --- Attachment: HADOOP-14451-04.patch > Deadlock in NativeIO > > > Key: HADOOP-14451 > URL: https://issues.apache.org/jira/browse/HADOOP-14451 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Vinayakumar B >Priority: Blocker > Attachments: HADOOP-14451-01.patch, HADOOP-14451-02.patch, > HADOOP-14451-03.patch, HADOOP-14451-04.patch, Nodemanager.jstack > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14832) Listing s3a bucket without credentials gives Interrupted error
[ https://issues.apache.org/jira/browse/HADOOP-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14832. - Resolution: Cannot Reproduce > Listing s3a bucket without credentials gives Interrupted error > -- > > Key: HADOOP-14832 > URL: https://issues.apache.org/jira/browse/HADOOP-14832 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: John Zhuge >Priority: Minor > > In trunk pseudo distributed mode, without setting s3a credentials, listing an > s3a bucket only gives "Interrupted" error : > {noformat} > $ hadoop fs -ls s3a://bucket/ > ls: Interrupted > {noformat} > In comparison, branch-2 gives a much better error message: > {noformat} > (branch-2)$ hadoop_env hadoop fs -ls s3a://bucket/ > ls: doesBucketExist on hdfs-cce: com.amazonaws.AmazonClientException: No AWS > Credentials provided by BasicAWSCredentialsProvider > EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : > com.amazonaws.SdkClientException: Unable to load credentials from service > endpoint > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12502) SetReplication OutOfMemoryError
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-12502: --- Attachment: HADOOP-12502-10.patch > SetReplication OutOfMemoryError > --- > > Key: HADOOP-12502 > URL: https://issues.apache.org/jira/browse/HADOOP-12502 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Philipp Schuegerl >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-12502-01.patch, HADOOP-12502-02.patch, > HADOOP-12502-03.patch, HADOOP-12502-04.patch, HADOOP-12502-05.patch, > HADOOP-12502-06.patch, HADOOP-12502-07.patch, HADOOP-12502-08.patch, > HADOOP-12502-09.patch, HADOOP-12502-10.patch > > > Setting the replication of a HDFS folder recursively can run out of memory. > E.g. with a large /var/log directory: > hdfs dfs -setrep -R -w 1 /var/log > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit > exceeded > at java.util.Arrays.copyOfRange(Arrays.java:2694) > at java.lang.String.(String.java:203) > at java.lang.String.substring(String.java:1913) > at java.net.URI$Parser.substring(URI.java:2850) > at java.net.URI$Parser.parse(URI.java:3046) > at java.net.URI.(URI.java:753) > at org.apache.hadoop.fs.Path.initialize(Path.java:203) > at org.apache.hadoop.fs.Path.(Path.java:116) > at org.apache.hadoop.fs.Path.(Path.java:94) > at > org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:222) > at > org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:246) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:689) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) > at > org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712) > at > org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708) > at > org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:268) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244) > at > org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:76) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363668#comment-16363668 ] Vinayakumar B commented on HADOOP-12502: Updated the patch. Please review. > SetReplication OutOfMemoryError > --- > > Key: HADOOP-12502 > URL: https://issues.apache.org/jira/browse/HADOOP-12502 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Philipp Schuegerl >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-12502-01.patch, HADOOP-12502-02.patch, > HADOOP-12502-03.patch, HADOOP-12502-04.patch, HADOOP-12502-05.patch, > HADOOP-12502-06.patch, HADOOP-12502-07.patch, HADOOP-12502-08.patch, > HADOOP-12502-09.patch, HADOOP-12502-10.patch > > > Setting the replication of a HDFS folder recursively can run out of memory. > E.g. with a large /var/log directory: > hdfs dfs -setrep -R -w 1 /var/log > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit > exceeded > at java.util.Arrays.copyOfRange(Arrays.java:2694) > at java.lang.String.(String.java:203) > at java.lang.String.substring(String.java:1913) > at java.net.URI$Parser.substring(URI.java:2850) > at java.net.URI$Parser.parse(URI.java:3046) > at java.net.URI.(URI.java:753) > at org.apache.hadoop.fs.Path.initialize(Path.java:203) > at org.apache.hadoop.fs.Path.(Path.java:116) > at org.apache.hadoop.fs.Path.(Path.java:94) > at > org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:222) > at > org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:246) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:689) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) > at > org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712) > at > org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708) > at > org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:268) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244) > at > org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:76) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363672#comment-16363672 ] Vinayakumar B commented on HADOOP-14451: updated the patch. Unified the tests into single class. > Deadlock in NativeIO > > > Key: HADOOP-14451 > URL: https://issues.apache.org/jira/browse/HADOOP-14451 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Vinayakumar B >Priority: Blocker > Attachments: HADOOP-14451-01.patch, HADOOP-14451-02.patch, > HADOOP-14451-03.patch, HADOOP-14451-04.patch, Nodemanager.jstack > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[ https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15227. - Resolution: Invalid > add mapreduce.outputcommitter.factory.scheme.s3a to core-default > > > Key: HADOOP-15227 > URL: https://issues.apache.org/jira/browse/HADOOP-15227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > > Need to add this property to core-default.xml. It's documented as being > there, but it isn't. > {code} > > mapreduce.outputcommitter.factory.scheme.s3a > org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory > > The committer factory to use when writing data to S3A filesystems. > > > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[ https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363783#comment-16363783 ] Steve Loughran commented on HADOOP-15227: - oh, maybe it's mapred default. Explains why I couldn't find it. I'm implementing a filesystem diagnostics JAR in https://github.com/steveloughran/cloudstore ; it wasn't finding it. But if it doesn't get loaded until JobConf is loaded then yes, it'll be there. In which case: docs are wrong. Let me fix them instead. And I'll force load mapred-default & site through a bit of reflection if is thsre > add mapreduce.outputcommitter.factory.scheme.s3a to core-default > > > Key: HADOOP-15227 > URL: https://issues.apache.org/jira/browse/HADOOP-15227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > > Need to add this property to core-default.xml. It's documented as being > there, but it isn't. > {code} > > mapreduce.outputcommitter.factory.scheme.s3a > org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory > > The committer factory to use when writing data to S3A filesystems. > > > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15076) Enhance s3a troubleshooting docs, add perf section
[ https://issues.apache.org/jira/browse/HADOOP-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363773#comment-16363773 ] Steve Loughran commented on HADOOP-15076: - Reviews please! > Enhance s3a troubleshooting docs, add perf section > -- > > Key: HADOOP-15076 > URL: https://issues.apache.org/jira/browse/HADOOP-15076 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15076-001.patch, HADOOP-15076-002.patch, > HADOOP-15076-003.patch, HADOOP-15076-004.patch, HADOOP-15076-005.patch, > HADOOP-15076-006.patch > > > A recurrent theme in s3a-related JIRAs, support calls etc is "tried upgrading > the AWS SDK JAR and then I got the error ...". We know here "don't do that", > but its not something immediately obvious to lots of downstream users who > want to be able to drop in the new JAR to fix things/add new features > We need to spell this out quite clearlyi "you cannot safely expect to do > this. If you want to upgrade the SDK, you will need to rebuild the whole of > hadoop-aws with the maven POM updated to the latest version, ideally > rerunning all the tests to make sure something hasn't broken. > Maybe near the top of the index.md file, along with "never share your AWS > credentials with anyone" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15232) AWSCredentialProviderList to throw custom NoCredentialsException; retry logic to handle
Steve Loughran created HADOOP-15232: --- Summary: AWSCredentialProviderList to throw custom NoCredentialsException; retry logic to handle Key: HADOOP-15232 URL: https://issues.apache.org/jira/browse/HADOOP-15232 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0 Reporter: Steve Loughran When no credentials are found in the list, an {{AmazonClientException}} is raised with a message, which may then be retried the S3a retry code. If a specific exception were raised, the retry handler would know to skip everything here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363787#comment-16363787 ] Steve Loughran commented on HADOOP-15107: - fix up docs to note the committer bindings are in mapred-default.xml > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. > I'm now not sure that the magic committer works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15232) AWSCredentialProviderList to throw custom NoCredentialsException; retry logic to handle
[ https://issues.apache.org/jira/browse/HADOOP-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363770#comment-16363770 ] Steve Loughran commented on HADOOP-15232: - Stack {code}2018-02-14 10:43:31,663 [main] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(127)) - No credentials provided by com.amazonaws.auth.InstanceProfileCredentialsProvider@6aa8e115: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141) at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:117) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1163) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4990) at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4964) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4213) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:366) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:365) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:301) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:249) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:232) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104) at org.apache.hadoop.fs.shell.Command.run(Command.java:176) at org.apache.hadoop.fs.FsShell.run(FsShell.java:328) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:391) Caused by: java.net.ConnectException: Network is unreachable (connect failed) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at
[jira] [Commented] (HADOOP-14451) Deadlock in NativeIO
[ https://issues.apache.org/jira/browse/HADOOP-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363796#comment-16363796 ] genericqa commented on HADOOP-14451: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 8s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 7s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 41s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 86m 55s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.http.TestHttpServerWithSpengo | | | hadoop.security.token.delegation.web.TestWebDelegationToken | | | hadoop.log.TestLogLevel | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910549/HADOOP-14451-04.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux cb9ad4ac7418 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 60971b8 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/14114/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14114/testReport/ | | Max. process+thread count | 1398 (vs. ulimit of 5500) | | modules
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364188#comment-16364188 ] Steve Loughran commented on HADOOP-15176: - javac are about use of deprecated AWS SDK methods; same issue as before: SDK moving too fast. checkstyle are about public fields in a structure which is marshalled to/from JSON I don't have any more to add here > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14707: Status: Patch Available (was: Open) > AbstractContractDistCpTest to test attr preservation with -p, verify > blobstores downgrade > - > > Key: HADOOP-14707 > URL: https://issues.apache.org/jira/browse/HADOOP-14707 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, test, tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch, > HADOOP-14707-003.patch > > > It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace > {code} > java.lang.UnsupportedOperationException: S3AFileSystem doesn't support > getXAttrs > at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559) > at > org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322) > > {code} > Add a test to {{AbstractContractDistCpTest}} to verify that this is handled > better. What is "handle better" here? Either ignore the option or fail with > "don't do that" text -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364276#comment-16364276 ] Steve Loughran commented on HADOOP-14707: - Patch 003: * FileContext has the same method, which now adds an IOE to aid its use there. * Described in filesystem.md As the same method is in FS and FC, worthwhile considering making an interface > AbstractContractDistCpTest to test attr preservation with -p, verify > blobstores downgrade > - > > Key: HADOOP-14707 > URL: https://issues.apache.org/jira/browse/HADOOP-14707 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, test, tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch, > HADOOP-14707-003.patch > > > It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace > {code} > java.lang.UnsupportedOperationException: S3AFileSystem doesn't support > getXAttrs > at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559) > at > org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322) > > {code} > Add a test to {{AbstractContractDistCpTest}} to verify that this is handled > better. What is "handle better" here? Either ignore the option or fail with > "don't do that" text -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15208: Status: Patch Available (was: Open) > DistCp to offer option to save src/dest filesets as alternative to delete() > --- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14764) Über-jira adl:// Azure Data Lake Phase II: Performance, Resilience and Testing
[ https://issues.apache.org/jira/browse/HADOOP-14764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14764: Affects Version/s: (was: 2.8.0) 3.0.0 > Über-jira adl:// Azure Data Lake Phase II: Performance, Resilience and Testing > -- > > Key: HADOOP-14764 > URL: https://issues.apache.org/jira/browse/HADOOP-14764 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > > Uber-JIRA for adl:// phase II > * Split out integration tests > * Parallel test execution > * More metrics > * Performance optimizations > * Performance tuning docs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14707: Target Version/s: 3.2.0 (was: 3.1.0) > AbstractContractDistCpTest to test attr preservation with -p, verify > blobstores downgrade > - > > Key: HADOOP-14707 > URL: https://issues.apache.org/jira/browse/HADOOP-14707 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, test, tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch > > > It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace > {code} > java.lang.UnsupportedOperationException: S3AFileSystem doesn't support > getXAttrs > at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559) > at > org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322) > > {code} > Add a test to {{AbstractContractDistCpTest}} to verify that this is handled > better. What is "handle better" here? Either ignore the option or fail with > "don't do that" text -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15208: Status: Open (was: Patch Available) > DistCp to offer option to save src/dest filesets as alternative to delete() > --- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14764) Über-jira adl:// Azure Data Lake Phase II: Performance, Resilience and Testing
[ https://issues.apache.org/jira/browse/HADOOP-14764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14764: Target Version/s: 3.2.0 (was: 2.8.3) > Über-jira adl:// Azure Data Lake Phase II: Performance, Resilience and Testing > -- > > Key: HADOOP-14764 > URL: https://issues.apache.org/jira/browse/HADOOP-14764 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > > Uber-JIRA for adl:// phase II > * Split out integration tests > * Parallel test execution > * More metrics > * Performance optimizations > * Performance tuning docs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15208: Status: Patch Available (was: Open) > DistCp to offer option to save src/dest filesets as alternative to delete() > --- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15208: Attachment: HADOOP-15208-002.patch > DistCp to offer option to save src/dest filesets as alternative to delete() > --- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[ https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364291#comment-16364291 ] Steve Loughran commented on HADOOP-15227: - I'd done that for HDFS as that's always in my head. Mapred conf is the solution here, though I may think about using a bit of reflection so if the jar isn't on the CP, nothing fails. > add mapreduce.outputcommitter.factory.scheme.s3a to core-default > > > Key: HADOOP-15227 > URL: https://issues.apache.org/jira/browse/HADOOP-15227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > > Need to add this property to core-default.xml. It's documented as being > there, but it isn't. > {code} > > mapreduce.outputcommitter.factory.scheme.s3a > org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory > > The committer factory to use when writing data to S3A filesystems. > > > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13972: Status: Patch Available (was: Open) > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364265#comment-16364265 ] Wangda Tan commented on HADOOP-15040: - [~fabbri], I saw this target version set to 3.1.0 but this is not in branch-3.1, could you double check this? And is there any other tickets should be pushed to branch-3.1? > Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation > --- > > Key: HADOOP-15040 > URL: https://issues.apache.org/jira/browse/HADOOP-15040 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Fix For: 3.1.0, 3.0.2 > > Attachments: HADOOP-15040.001.patch > > > My colleagues working with Yarn log aggregation found that they were getting > this message spammed in their logs when they used an s3a:// URI for logs > (yarn.nodemanager.remote-app-log-dir): > {noformat} > getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw > an exception > javax.management.RuntimeMBeanException: java.lang.NullPointerException > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839) > at > > Caused by: java.lang.NullPointerException > at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729) > at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67) > at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > {noformat} > This happens even though the aws sdk cloudwatch metrics reporting was > disabled (default), which is a bug. > I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and > it looks like a fix should be coming around SDK release 1.11.229 or so. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[ https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364275#comment-16364275 ] Jason Lowe commented on HADOOP-15227: - Yeah, mapred-default and mapred-site aren't loaded until the JobConf class is loaded. A common mistake for code is to create a plain Configuration early in {{main}} and try to lookup mapred properties (or even hdfs or yarn properties) expecting to get the default if they are not set by the user. The easy fix is to create a JobConf instead of a Configuration if the code knows it wants to do mapred stuff. > add mapreduce.outputcommitter.factory.scheme.s3a to core-default > > > Key: HADOOP-15227 > URL: https://issues.apache.org/jira/browse/HADOOP-15227 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > > Need to add this property to core-default.xml. It's documented as being > there, but it isn't. > {code} > > mapreduce.outputcommitter.factory.scheme.s3a > org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory > > The committer factory to use when writing data to S3A filesystems. > > > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14707: Status: Open (was: Patch Available) > AbstractContractDistCpTest to test attr preservation with -p, verify > blobstores downgrade > - > > Key: HADOOP-14707 > URL: https://issues.apache.org/jira/browse/HADOOP-14707 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, test, tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch > > > It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace > {code} > java.lang.UnsupportedOperationException: S3AFileSystem doesn't support > getXAttrs > at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559) > at > org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322) > > {code} > Add a test to {{AbstractContractDistCpTest}} to verify that this is handled > better. What is "handle better" here? Either ignore the option or fail with > "don't do that" text -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14707: Attachment: HADOOP-14707-003.patch > AbstractContractDistCpTest to test attr preservation with -p, verify > blobstores downgrade > - > > Key: HADOOP-14707 > URL: https://issues.apache.org/jira/browse/HADOOP-14707 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, test, tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch, > HADOOP-14707-003.patch > > > It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace > {code} > java.lang.UnsupportedOperationException: S3AFileSystem doesn't support > getXAttrs > at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559) > at > org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322) > > {code} > Add a test to {{AbstractContractDistCpTest}} to verify that this is handled > better. What is "handle better" here? Either ignore the option or fail with > "don't do that" text -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15233) PathData invalid relativize for root and root dir
George Smith created HADOOP-15233: - Summary: PathData invalid relativize for root and root dir Key: HADOOP-15233 URL: https://issues.apache.org/jira/browse/HADOOP-15233 Project: Hadoop Common Issue Type: Bug Components: common, fs, tools Affects Versions: 3.0.0, 2.7.3 Reporter: George Smith Attachments: stacktrace.txt Let's say current working directory is set to "/" (FileSystem.setCurrentWorkingDirectory()) and there is "/test" folder on HDFS. when I call {code:java} org.apache.hadoop.fs.shell.PathData.expandAsGlob("test", new Configuration()); {code} (the relative path to _test_ is an intention) then I get PathData with a "../test" path. But such path is invalid. The problem is probably inside Path.relativize method, where for "/" and "test" the "../test" is produced. What I really want is to call "*rm -R test*" command with cwd set to root ("/"), but it won't work because command expands it to invalid path (DFSUtil.isValidName(result) is false). Using relative paths will work fine for another cwd except the root. This is just the only special case for which it fails. Stacktrace attached for more details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[ https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364386#comment-16364386 ] Hudson commented on HADOOP-10571: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13655 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13655/]) HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. (stevel: rev f20dc0d5770d3876954faf0a6e8dcce6539ffc23) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/TestServiceOperations.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/AbstractService.java * (edit) hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * (edit) hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/HttpInputStreamWithRelease.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryUtils.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java * (edit) hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyCheckpointer.java > Use Log.*(Object, Throwable) overload to log exceptions > --- > > Key: HADOOP-10571 > URL: https://issues.apache.org/jira/browse/HADOOP-10571 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Andras Bokor >Priority: Major > Attachments: HADOOP-10571.01.patch, HADOOP-10571.01.patch, > HADOOP-10571.02.patch, HADOOP-10571.03.patch, HADOOP-10571.04.patch, > HADOOP-10571.05.patch, HADOOP-10571.06.patch, HADOOP-10571.07.patch > > > When logging an exception, we often convert the exception to string or call > {{.getMessage}}. Instead we can use the log method overloads which take > {{Throwable}} as a parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364419#comment-16364419 ] ASF GitHub Bot commented on HADOOP-13972: - Github user ssonker commented on the issue: https://github.com/apache/hadoop/pull/339 @steveloughran Where can I find the build status of the patch. I was looking at https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/ but couldn't locate this patch. Thanks. > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14799) Update nimbus-jose-jwt to 4.41.1
[ https://issues.apache.org/jira/browse/HADOOP-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364549#comment-16364549 ] Brahma Reddy Battula commented on HADOOP-14799: --- [~rchiang]/[~lmccay] do you think any compatibility/down-stream issues will be there on upgrade..? this CVE got fixed in 4.39 version and we were using 3.9 version,so to address this CVE ,we've to upgrade. > Update nimbus-jose-jwt to 4.41.1 > > > Key: HADOOP-14799 > URL: https://issues.apache.org/jira/browse/HADOOP-14799 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1, 2.10.0, 2.9.1, 2.8.4 > > Attachments: HADOOP-14799.001.patch, HADOOP-14799.002.patch, > HADOOP-14799.003.patch > > > Update the dependency > com.nimbusds:nimbus-jose-jwt:3.9 > to the latest (4.41.1) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364496#comment-16364496 ] ASF GitHub Bot commented on HADOOP-13972: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/339 the results should be added as a JIRA comment once completed > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14903) Add json-smart explicitly to pom.xml
[ https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-14903: -- Attachment: HADOOP-14903-003-branch-2.patch > Add json-smart explicitly to pom.xml > > > Key: HADOOP-14903 > URL: https://issues.apache.org/jira/browse/HADOOP-14903 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14903-003-branch-2.patch, HADOOP-14903.001.patch, > HADOOP-14903.002.patch, HADOOP-14903.003.patch > > > With the library update in HADOOP-14799, maven knows how to pull in > net.minidev:json-smart for tests, but not for packaging. This needs to be > added to the main project pom in order to avoid this warning: > {noformat} > [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no > dependency information available > {noformat} > This is pulled in from a few places: > {noformat} > [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile > [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile > [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile > [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile > [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile > [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile > [INFO] | |+- > com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile > [INFO] | |\- net.minidev:json-smart:jar:2.3:compile > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15225) mvn javadoc:test-javadoc goal throws cannot find symbol
[ https://issues.apache.org/jira/browse/HADOOP-15225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor updated HADOOP-15225: -- Description: {code:java} hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestReflectionUtils.java:28: error: cannot find symbol [WARNING] import static org.hamcrest.CoreMatchers.containsString; [WARNING] ^ [WARNING] symbol: static containsString [WARNING] location: class{code} This happens because mockito-all includes Hamcrest classes but a different version. Let's see TestReflectionUtils as an example: {{import static org.hamcrest.CoreMatchers.containsString; }} will result in error. Somehow mvn javadoc:test-javadoc will find Mockito's CoreMatchers class on the classpath which has no containsString method. From Mockito 2 the mockito-all is discontinued so HADOOP-14178 will solve this. Once HADOOP-14178 is resolved this can be closed as well. was: {code:java} hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestReflectionUtils.java:28: error: cannot find symbol [WARNING] import static org.hamcrest.CoreMatchers.containsString; [WARNING] ^ [WARNING] symbol: static containsString [WARNING] location: class{code} This happens because mockito-all includes Hamcrest classes but a different version. Let's see TestReflectionUtils as an example: {{import static org.hamcrest.CoreMatchers.containsString; }}will result in error. Somehow mvn javadoc:test-javadoc will find Mockito's CoreMatchers class on the classpath which has no containsString method. From Mockito 2 the mockito-all is discontinued so HADOOP-14178 will solve this. Once HADOOP-14178 is resolved this can be closed as well. > mvn javadoc:test-javadoc goal throws cannot find symbol > --- > > Key: HADOOP-15225 > URL: https://issues.apache.org/jira/browse/HADOOP-15225 > Project: Hadoop Common > Issue Type: Bug >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Minor > > {code:java} > hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestReflectionUtils.java:28: > error: cannot find symbol > [WARNING] import static org.hamcrest.CoreMatchers.containsString; > [WARNING] ^ > [WARNING] symbol: static containsString > [WARNING] location: class{code} > This happens because mockito-all includes Hamcrest classes but a different > version. Let's see TestReflectionUtils as an example: > {{import static org.hamcrest.CoreMatchers.containsString; }} will result in > error. > Somehow mvn javadoc:test-javadoc will find Mockito's CoreMatchers class on > the classpath which has no containsString method. > From Mockito 2 the mockito-all is discontinued so HADOOP-14178 will solve > this. > Once HADOOP-14178 is resolved this can be closed as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-11717) Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth
[ https://issues.apache.org/jira/browse/HADOOP-11717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362347#comment-16362347 ] Brahma Reddy Battula edited comment on HADOOP-11717 at 2/14/18 6:10 PM: Nice work Guys!! We should upgrade nimbus-jose-jwt, as there [CVE|https://nvd.nist.gov/vuln/detail/CVE-2017-12973#vulnDescriptionTitle] present before 4.39 version. HADOOP-14799 raised to upgrade but it's not committed to branch-2 and branch-2.8 versions. was (Author: brahmareddy): Nice work Guys!! We should upgrade nimbus-jose-jwt, as there [CVE|https://nvd.nist.gov/vuln/detail/CVE-2017-12973#vulnDescriptionTitle] present before 4.39 version. HADOOP-14799 raised to upgrade but it's committed to branch-2 and branch-2.8 versions. > Add Redirecting WebSSO behavior with JWT Token in Hadoop Auth > - > > Key: HADOOP-11717 > URL: https://issues.apache.org/jira/browse/HADOOP-11717 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HADOOP-11717-1.patch, HADOOP-11717-2.patch, > HADOOP-11717-3.patch, HADOOP-11717-4.patch, HADOOP-11717-5.patch, > HADOOP-11717-6.patch, HADOOP-11717-7.patch, HADOOP-11717-8.patch, > RedirectingWebSSOwithJWTforHadoopWebUIs.pdf > > > Extend AltKerberosAuthenticationHandler to provide WebSSO flow for UIs. > The actual authentication is done by some external service that the handler > will redirect to when there is no hadoop.auth cookie and no JWT token found > in the incoming request. > Using JWT provides a number of benefits: > * It is not tied to any specific authentication mechanism - so buys us many > SSO integrations > * It is cryptographically verifiable for determining whether it can be trusted > * Checking for expiration allows for a limited lifetime and window for > compromised use > This will introduce the use of nimbus-jose-jwt library for processing, > validating and parsing JWT tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[ https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364357#comment-16364357 ] Steve Loughran commented on HADOOP-10571: - OK, +1 from me too committed to 3.1+ I'd support a backport to 3.0 too, for consistency across the 3.x line. It doesn't take right now with some conflicts in the nfs3 package. Options: fix the conflict (best), skip those files (workable) > Use Log.*(Object, Throwable) overload to log exceptions > --- > > Key: HADOOP-10571 > URL: https://issues.apache.org/jira/browse/HADOOP-10571 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Andras Bokor >Priority: Major > Attachments: HADOOP-10571.01.patch, HADOOP-10571.01.patch, > HADOOP-10571.02.patch, HADOOP-10571.03.patch, HADOOP-10571.04.patch, > HADOOP-10571.05.patch, HADOOP-10571.06.patch, HADOOP-10571.07.patch > > > When logging an exception, we often convert the exception to string or call > {{.getMessage}}. Instead we can use the log method overloads which take > {{Throwable}} as a parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15234) NPE when initializing KMSWebApp
Xiao Chen created HADOOP-15234: -- Summary: NPE when initializing KMSWebApp Key: HADOOP-15234 URL: https://issues.apache.org/jira/browse/HADOOP-15234 Project: Hadoop Common Issue Type: Bug Components: kms Reporter: Xiao Chen During KMS startup, if the {{keyProvider}} is null, it will NPE inside KeyProviderExtension. {noformat} java.lang.NullPointerException at org.apache.hadoop.crypto.key.KeyProviderExtension.(KeyProviderExtension.java:43) at org.apache.hadoop.crypto.key.CachingKeyProvider.(CachingKeyProvider.java:93) at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:170) {noformat} We're investigating the exact scenario that could lead to this, but the NPE and log around it can be improved. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364552#comment-16364552 ] Xiao Chen commented on HADOOP-12897: This appears to break several tests in other components https://builds.apache.org/job/PreCommit-HADOOP-Build/14118/testReport/ Please take a look. I'm reverting this for now. > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Fix For: 3.1.0, 3.0.1 > > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363858#comment-16363858 ] ASF GitHub Bot commented on HADOOP-13972: - Github user steveloughran commented on a diff in the pull request: https://github.com/apache/hadoop/pull/339#discussion_r168154574 --- Diff: hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java --- @@ -985,4 +995,65 @@ public void setUserGroupRepresentationAsUPN(boolean enableUPN) { oidOrUpn = enableUPN ? UserGroupRepresentation.UPN : UserGroupRepresentation.OID; } + + /** + * Gets ADL account name from ADL FQDN + * @param accountFQDN ADL account fqdn + * @return ADL account name + */ + public static String getAccountNameFromFQDN(String accountFQDN) { +return accountFQDN.contains(".") +? accountFQDN.substring(0, accountFQDN.indexOf(".")) +: accountFQDN; + } + + /** + * Propagates account-specific settings into generic ADL configuration keys. + * This is done by propagating the values of the form + * {@code fs.adl.account.${account_name}.key} to + * {@code fs.adl.key}, for all values of "key" + * + * The source of the updated property is set to the key name of the account + * property, to aid in diagnostics of where things came from. + * + * Returns a new configuration. Why the clone? + * You can use the same conf for different filesystems, and the original + * values are not updated. + * + * + * @param source Source Configuration object + * @param accountName account name. Must not be empty + * @return a (potentially) patched clone of the original + * @throws IOException + */ + public static Configuration propagateAccountOptions( + Configuration source, String accountName) throws IOException { +if (StringUtils.isEmpty(accountName)) { + throw new IOException("Account name cannot be empty"); --- End diff -- We're generally happy with {{Precondition.checkArgument()}} here, which raises IllegalArgumentException. Best just to use that, unless you have tests which look for the specific exception > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[ https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363876#comment-16363876 ] Andras Bokor commented on HADOOP-10571: --- I had to modify a wrong indented block so the 2 checkstyles are not caused by the patch. > Use Log.*(Object, Throwable) overload to log exceptions > --- > > Key: HADOOP-10571 > URL: https://issues.apache.org/jira/browse/HADOOP-10571 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Andras Bokor >Priority: Major > Attachments: HADOOP-10571.01.patch, HADOOP-10571.01.patch, > HADOOP-10571.02.patch, HADOOP-10571.03.patch, HADOOP-10571.04.patch, > HADOOP-10571.05.patch, HADOOP-10571.06.patch, HADOOP-10571.07.patch > > > When logging an exception, we often convert the exception to string or call > {{.getMessage}}. Instead we can use the log method overloads which take > {{Throwable}} as a parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363890#comment-16363890 ] fang zhenyi commented on HADOOP-15223: -- Thanks [~ajisakaa] for comments.I have remove all of \{{Collections.EMPTY_MAP}},\{{Collections.EMPTY_SET}} , \{{Collections.emptyList()}} and unnecessary @SuppressWarning("unchecked") annotations.Hope you can review again, thanks a lot. > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[ https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363890#comment-16363890 ] fang zhenyi edited comment on HADOOP-15223 at 2/14/18 12:58 PM: Thanks [~ajisakaa] for comments.I have remove all of {{Collections.EMPTY_MAP}},{{Collections.EMPTY_SET}} , {{Collections.emptyList()}} and unnecessary @SuppressWarning("unchecked") annotations.Hope you can review again, thanks a lot. was (Author: zhenyi): Thanks [~ajisakaa] for comments.I have remove all of \{{Collections.EMPTY_MAP}},\{{Collections.EMPTY_SET}} , \{{Collections.emptyList()}} and unnecessary @SuppressWarning("unchecked") annotations.Hope you can review again, thanks a lot. > Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() > when available > - > > Key: HADOOP-15223 > URL: https://issues.apache.org/jira/browse/HADOOP-15223 > Project: Hadoop Common > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: fang zhenyi >Priority: Minor > Labels: newbie > Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch > > > The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often > causes unchecked assignment and it should be replaced with > {{Collections.emptySet()}} and {{Collections.emptyMap()}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15208: Status: Patch Available (was: Open) > DistCp to offer option to save src/dest filesets as alternative to delete() > --- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch, HADOOP-15208-003.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14799) Update nimbus-jose-jwt to 4.41.1
[ https://issues.apache.org/jira/browse/HADOOP-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-14799: Fix Version/s: (was: 2.8.4) (was: 2.9.1) (was: 2.10.0) > Update nimbus-jose-jwt to 4.41.1 > > > Key: HADOOP-14799 > URL: https://issues.apache.org/jira/browse/HADOOP-14799 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14799.001.patch, HADOOP-14799.002.patch, > HADOOP-14799.003.patch > > > Update the dependency > com.nimbusds:nimbus-jose-jwt:3.9 > to the latest (4.41.1) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[ https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364761#comment-16364761 ] Anu Engineer commented on HADOOP-15204: --- [~chris.douglas] / [~ste...@apache.org] Please let me know if you have any more comments. If this looks good, I will make corresponding changes in Ozone branch to use this feature. Thank you for the time and comments. > Add Configuration API for parsing storage sizes > --- > > Key: HADOOP-15204 > URL: https://issues.apache.org/jira/browse/HADOOP-15204 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.1.0 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Fix For: 3.1.0 > > Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch, > HADOOP-15204.003.patch > > > Hadoop has a lot of configurations that specify memory and disk size. This > JIRA proposes to add an API like {{Configuration.getStorageSize}} which will > allow users > to specify units like KB, MB, GB etc. This is JIRA is inspired by > HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great > improvement for ozone code base, this JIRA hopes to do the same thing for > configs that deal with disk and memory usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[ https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364772#comment-16364772 ] Chris Douglas commented on HADOOP-15204: +1 lgtm > Add Configuration API for parsing storage sizes > --- > > Key: HADOOP-15204 > URL: https://issues.apache.org/jira/browse/HADOOP-15204 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.1.0 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Fix For: 3.1.0 > > Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch, > HADOOP-15204.003.patch > > > Hadoop has a lot of configurations that specify memory and disk size. This > JIRA proposes to add an API like {{Configuration.getStorageSize}} which will > allow users > to specify units like KB, MB, GB etc. This is JIRA is inspired by > HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great > improvement for ozone code base, this JIRA hopes to do the same thing for > configs that deal with disk and memory usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14707: Status: Open (was: Patch Available) > AbstractContractDistCpTest to test attr preservation with -p, verify > blobstores downgrade > - > > Key: HADOOP-14707 > URL: https://issues.apache.org/jira/browse/HADOOP-14707 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, test, tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch, > HADOOP-14707-003.patch > > > It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace > {code} > java.lang.UnsupportedOperationException: S3AFileSystem doesn't support > getXAttrs > at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559) > at > org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322) > > {code} > Add a test to {{AbstractContractDistCpTest}} to verify that this is handled > better. What is "handle better" here? Either ignore the option or fail with > "don't do that" text -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14799) Update nimbus-jose-jwt to 4.41.1
[ https://issues.apache.org/jira/browse/HADOOP-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364800#comment-16364800 ] Kihwal Lee commented on HADOOP-14799: - A bunch of tests started failing in branch-2 through branch-2.8 with this commit. Reverted. E.g. TestSymlinkHdfsFileSystem > Update nimbus-jose-jwt to 4.41.1 > > > Key: HADOOP-14799 > URL: https://issues.apache.org/jira/browse/HADOOP-14799 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1, 2.10.0, 2.9.1, 2.8.4 > > Attachments: HADOOP-14799.001.patch, HADOOP-14799.002.patch, > HADOOP-14799.003.patch > > > Update the dependency > com.nimbusds:nimbus-jose-jwt:3.9 > to the latest (4.41.1) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[ https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-15206: Status: Patch Available (was: Open) Thanks for updating the patch! Looks good overall, just a few nits. I think we're close, so moving this to Patch Available so the QA bot can comment on this as well. Why are we only skipping one byte at a time instead of trying to skip the rest of the way in one call? The code can track the remaining bytes in skipBytes, decrement that by the number of bytes skipped in the loop, then loop while skipBytes > 0. There is trailing whitespace on a couple of lines which would be nice to cleanup. I expect the QA bot to flag this in its whitespace check. I'm not sure it's necessary to call out the JIRA in the comments. That's what {{git blame}} is for. ;) Otherwise the code would be littered with JIRA numbers in every bugfix change. "steam is on BZip2 header" should be "a split is before the first BZip2 block" > BZip2 drops and duplicates records when input split size is small > - > > Key: HADOOP-15206 > URL: https://issues.apache.org/jira/browse/HADOOP-15206 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0, 2.8.3 >Reporter: Aki Tanaka >Priority: Major > Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch, > HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch, > HADOOP-15206.005.patch, HADOOP-15206.006.patch > > > BZip2 can drop and duplicate record when input split file is small. I > confirmed that this issue happens when the input split size is between 1byte > and 4bytes. > I am seeing the following 2 problem behaviors. > > 1. Drop record: > BZip2 skips the first record in the input file when the input split size is > small > > Set the split size to 3 and tested to load 100 records (0, 1, 2..99) > {code:java} > 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(317)) - > splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3 > count=99{code} > > The input format read only 99 records but not 100 records > > 2. Duplicate Record: > 2 input splits has same BZip2 records when the input split size is small > > Set the split size to 1 and tested to load 100 records (0, 1, 2..99) > > {code:java} > 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file > /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1 > count=99 > 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4 > at position 8 > {code} > > I experienced this error when I execute Spark (SparkSQL) job under the > following conditions: > * The file size of the input files are small (around 1KB) > * Hadoop cluster has many slave nodes (able to launch many executor tasks) > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15076) Enhance s3a troubleshooting docs, add perf section
[ https://issues.apache.org/jira/browse/HADOOP-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15076: Status: Open (was: Patch Available) one more stack trace to come; remembered to add it. The "integrity" error related to signing > Enhance s3a troubleshooting docs, add perf section > -- > > Key: HADOOP-15076 > URL: https://issues.apache.org/jira/browse/HADOOP-15076 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15076-001.patch, HADOOP-15076-002.patch, > HADOOP-15076-003.patch, HADOOP-15076-004.patch, HADOOP-15076-005.patch, > HADOOP-15076-006.patch > > > A recurrent theme in s3a-related JIRAs, support calls etc is "tried upgrading > the AWS SDK JAR and then I got the error ...". We know here "don't do that", > but its not something immediately obvious to lots of downstream users who > want to be able to drop in the new JAR to fix things/add new features > We need to spell this out quite clearlyi "you cannot safely expect to do > this. If you want to upgrade the SDK, you will need to rebuild the whole of > hadoop-aws with the maven POM updated to the latest version, ideally > rerunning all the tests to make sure something hasn't broken. > Maybe near the top of the index.md file, along with "never share your AWS > credentials with anyone" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15235) Authentication Tokens should use HMAC instead of MAC
[ https://issues.apache.org/jira/browse/HADOOP-15235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-15235: --- Status: Patch Available (was: Open) > Authentication Tokens should use HMAC instead of MAC > > > Key: HADOOP-15235 > URL: https://issues.apache.org/jira/browse/HADOOP-15235 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.10.0, 3.2.0 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: HADOOP-15235.001.patch, HADOOP-15235.002.patch > > > We currently use {{MessageDigest}} to compute a "SHA" MAC for signing > Authentication Tokens. Firstly, what "SHA" maps to is dependent on the JVM > and Cryptography Provider. While they _should_ do something reasonable, it's > probably a safer idea to pick a specific algorithm. It looks like the Oracle > JVM picks SHA-1; though something like SHA-256 would be better. > In any case, it would also be better to use an HMAC algorithm instead. > Changing from SHA-1 to SHA-256 or MAC to HMAC won't generate equivalent > signatures, so this would normally be an incompatible change because the > server wouldn't accept previous tokens it issued with the older algorithm. > However, Authentication Tokens are used as a cheaper shortcut for Kerberos, > so it's expected for users to also have Kerberos credentials; in this case, > the Authentication Token will be rejected, but it will silently retry using > Kerberos, and get an updated token. So this should all be transparent to the > user. > And finally, the code where we verify a signature uses a non-constant-time > comparison, which could be subject to timing attacks. I believe it would be > quite difficult in this case to do so, but we're probably better off using a > constant-time comparison. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15235) Authentication Tokens should use HMAC instead of MAC
[ https://issues.apache.org/jira/browse/HADOOP-15235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364906#comment-16364906 ] Robert Kanter commented on HADOOP-15235: The 002 patch replaces the call to {{Base64(0).encodeAsString(sig);}} with {{Base64(0).encodeToString(sig);}}, which works in branch-2 and branch-3. > Authentication Tokens should use HMAC instead of MAC > > > Key: HADOOP-15235 > URL: https://issues.apache.org/jira/browse/HADOOP-15235 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.10.0, 3.2.0 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: HADOOP-15235.001.patch, HADOOP-15235.002.patch > > > We currently use {{MessageDigest}} to compute a "SHA" MAC for signing > Authentication Tokens. Firstly, what "SHA" maps to is dependent on the JVM > and Cryptography Provider. While they _should_ do something reasonable, it's > probably a safer idea to pick a specific algorithm. It looks like the Oracle > JVM picks SHA-1; though something like SHA-256 would be better. > In any case, it would also be better to use an HMAC algorithm instead. > Changing from SHA-1 to SHA-256 or MAC to HMAC won't generate equivalent > signatures, so this would normally be an incompatible change because the > server wouldn't accept previous tokens it issued with the older algorithm. > However, Authentication Tokens are used as a cheaper shortcut for Kerberos, > so it's expected for users to also have Kerberos credentials; in this case, > the Authentication Token will be rejected, but it will silently retry using > Kerberos, and get an updated token. So this should all be transparent to the > user. > And finally, the code where we verify a signature uses a non-constant-time > comparison, which could be subject to timing attacks. I believe it would be > quite difficult in this case to do so, but we're probably better off using a > constant-time comparison. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15235) Authentication Tokens should use HMAC instead of MAC
[ https://issues.apache.org/jira/browse/HADOOP-15235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-15235: --- Attachment: HADOOP-15235.002.patch > Authentication Tokens should use HMAC instead of MAC > > > Key: HADOOP-15235 > URL: https://issues.apache.org/jira/browse/HADOOP-15235 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.10.0, 3.2.0 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: HADOOP-15235.001.patch, HADOOP-15235.002.patch > > > We currently use {{MessageDigest}} to compute a "SHA" MAC for signing > Authentication Tokens. Firstly, what "SHA" maps to is dependent on the JVM > and Cryptography Provider. While they _should_ do something reasonable, it's > probably a safer idea to pick a specific algorithm. It looks like the Oracle > JVM picks SHA-1; though something like SHA-256 would be better. > In any case, it would also be better to use an HMAC algorithm instead. > Changing from SHA-1 to SHA-256 or MAC to HMAC won't generate equivalent > signatures, so this would normally be an incompatible change because the > server wouldn't accept previous tokens it issued with the older algorithm. > However, Authentication Tokens are used as a cheaper shortcut for Kerberos, > so it's expected for users to also have Kerberos credentials; in this case, > the Authentication Token will be rejected, but it will silently retry using > Kerberos, and get an updated token. So this should all be transparent to the > user. > And finally, the code where we verify a signature uses a non-constant-time > comparison, which could be subject to timing attacks. I believe it would be > quite difficult in this case to do so, but we're probably better off using a > constant-time comparison. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[ https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned HADOOP-15206: --- Assignee: Aki Tanaka > BZip2 drops and duplicates records when input split size is small > - > > Key: HADOOP-15206 > URL: https://issues.apache.org/jira/browse/HADOOP-15206 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.3, 3.0.0 >Reporter: Aki Tanaka >Assignee: Aki Tanaka >Priority: Major > Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch, > HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch, > HADOOP-15206.005.patch, HADOOP-15206.006.patch > > > BZip2 can drop and duplicate record when input split file is small. I > confirmed that this issue happens when the input split size is between 1byte > and 4bytes. > I am seeing the following 2 problem behaviors. > > 1. Drop record: > BZip2 skips the first record in the input file when the input split size is > small > > Set the split size to 3 and tested to load 100 records (0, 1, 2..99) > {code:java} > 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(317)) - > splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3 > count=99{code} > > The input format read only 99 records but not 100 records > > 2. Duplicate Record: > 2 input splits has same BZip2 records when the input split size is small > > Set the split size to 1 and tested to load 100 records (0, 1, 2..99) > > {code:java} > 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file > /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1 > count=99 > 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4 > at position 8 > {code} > > I experienced this error when I execute Spark (SparkSQL) job under the > following conditions: > * The file size of the input files are small (around 1KB) > * Hadoop cluster has many slave nodes (able to launch many executor tasks) > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364925#comment-16364925 ] Aaron Fabbri commented on HADOOP-15040: --- I did not realize there was also a 3.1 branch.. I will commit to that as well and update fix version here when it is done. > Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation > --- > > Key: HADOOP-15040 > URL: https://issues.apache.org/jira/browse/HADOOP-15040 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Fix For: 3.1.0, 3.0.2 > > Attachments: HADOOP-15040.001.patch > > > My colleagues working with Yarn log aggregation found that they were getting > this message spammed in their logs when they used an s3a:// URI for logs > (yarn.nodemanager.remote-app-log-dir): > {noformat} > getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw > an exception > javax.management.RuntimeMBeanException: java.lang.NullPointerException > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839) > at > > Caused by: java.lang.NullPointerException > at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729) > at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67) > at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > {noformat} > This happens even though the aws sdk cloudwatch metrics reporting was > disabled (default), which is a bug. > I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and > it looks like a fix should be coming around SDK release 1.11.229 or so. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9477) Add posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-9477: Fix Version/s: 2.7.6 Just pushed this into branch-2.7. Updating fix version. > Add posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Dapeng Sun >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.004.patch, > HADOOP-9477.005.patch, HADOOP-9477.006.patch, HADOOP-9477.007.patch, > HADOOP-9477.008.patch, HADOOP-9477.009.patch, HADOOP-9477.patch, > HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364951#comment-16364951 ] genericqa commented on HADOOP-15208: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 8 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 33s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 7s{color} | {color:orange} root: The patch generated 1 new + 287 unchanged - 36 fixed = 288 total (was 323) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 51s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 41s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 39s{color} | {color:green} hadoop-distcp in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}125m 29s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-15208 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910615/HADOOP-15208-003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux afbd828a2127 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool |
[jira] [Updated] (HADOOP-12568) Update core-default.xml to describe posixGroups support
[ https://issues.apache.org/jira/browse/HADOOP-12568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-12568: - Fix Version/s: 2.7.6 Just pushed this into branch-2.7. Updating fix version. > Update core-default.xml to describe posixGroups support > --- > > Key: HADOOP-12568 > URL: https://issues.apache.org/jira/browse/HADOOP-12568 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Labels: group, mappings, supportability > Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6 > > Attachments: HADOOP-12568.001.patch, HADOOP-12568.002.patch > > > After HADOOP-9477, LdapGroupsMapping supports posixGroups mapping service. > However, core-default.xml was not updated to detail how to configure in order > to enable this feature. This JIRA is filed to describe how to enable > posixGroups for users. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364408#comment-16364408 ] genericqa commented on HADOOP-15208: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 0s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 36s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 41s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 18s{color} | {color:orange} root: The patch generated 6 new + 287 unchanged - 36 fixed = 293 total (was 323) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 12s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 9s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m 10s{color} | {color:green} hadoop-distcp in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}121m 8s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.http.TestHttpServerWithSpengo | | | hadoop.security.token.delegation.web.TestWebDelegationToken | | | hadoop.log.TestLogLevel | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-15208 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909668/HADOOP-15208-002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 9b55aeef492e 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64
[jira] [Commented] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[ https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364501#comment-16364501 ] genericqa commented on HADOOP-14707: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 55s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 56s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s{color} | {color:red} hadoop-azure in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s{color} | {color:red} hadoop-azure-datalake in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 41s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 41s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 34s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 21s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 18s{color} | {color:red} hadoop-azure in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 24s{color} | {color:red} hadoop-azure-datalake in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m 50s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 29s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 25s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 30s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s{color} |
[jira] [Commented] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364528#comment-16364528 ] genericqa commented on HADOOP-15208: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 16s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 20s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 10s{color} | {color:orange} root: The patch generated 6 new + 287 unchanged - 36 fixed = 293 total (was 323) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 30s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 4s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 46s{color} | {color:green} hadoop-distcp in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}105m 25s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.http.TestHttpServerWithSpengo | | | hadoop.log.TestLogLevel | | | hadoop.security.token.delegation.web.TestWebDelegationToken | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-15208 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910602/HADOOP-15208-002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux b99835d7353e 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64
[jira] [Updated] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[ https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14943: Status: Patch Available (was: Open) > Add common getFileBlockLocations() emulation for object stores, including S3A > - > > Key: HADOOP-14943 > URL: https://issues.apache.org/jira/browse/HADOOP-14943 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch, > HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch > > > It looks suspiciously like S3A isn't providing the partitioning data needed > in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a > file by the blocksize. This will stop tools using the MRv1 APIS doing the > partitioning properly if the input format isn't doing it own split logic. > FileInputFormat in MRv2 is a bit more configurable about input split > calculation & will split up large files. but otherwise, the partitioning is > being done more by the default values of the executing engine, rather than > any config data from the filesystem about what its "block size" is, > NativeAzureFS does a better job; maybe that could be factored out to > hadoop-common and reused? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14630: Status: Open (was: Patch Available) > Contract Tests to verify create, mkdirs and rename under a file is forbidden > > > Key: HADOOP-14630 > URL: https://issues.apache.org/jira/browse/HADOOP-14630 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, fs/swift >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch, > HADOOP-14630-003.patch > > > Object stores can get into trouble in ways which an FS would never, do, ways > so obvious we've never done tests for them. We know what the problems are: > test for file and dir creation directly/indirectly under other files > * mkdir(file/file) > * mkdir(file/subdir) > * dir under file/subdir/subdir > * dir/dir2/file, verify dir & dir2 exist > * dir/dir2/dir3, verify dir & dir2 exist > * rename(src, file/dest) > * rename(src, file/dir/dest) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-12897: Status: Patch Available (was: Reopened) > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch, > HADOOP-12897.006.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-12897: Attachment: HADOOP-12897.006.patch > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch, > HADOOP-12897.006.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364710#comment-16364710 ] Ajay Kumar commented on HADOOP-12897: - [~xiaochen],[~ste...@apache.org] Submitting patch v6 which handles failures from [https://builds.apache.org/job/Hadoop-trunk-Commit/13656/]. > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch, > HADOOP-12897.006.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[ https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364706#comment-16364706 ] Steve Loughran commented on HADOOP-14943: - if you return a specific host for the data, then it reports to the scheduler the preferred location of the work...the schedulers will try and place the work there and wait a bit before giving up. What you are measuring there is how long spark waits before rescheduling You don't want location affinity in object stores, not really ... though [~ehiggs] and [~Thomas Demoor] might have different data > Add common getFileBlockLocations() emulation for object stores, including S3A > - > > Key: HADOOP-14943 > URL: https://issues.apache.org/jira/browse/HADOOP-14943 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch, > HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch > > > It looks suspiciously like S3A isn't providing the partitioning data needed > in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a > file by the blocksize. This will stop tools using the MRv1 APIS doing the > partitioning properly if the input format isn't doing it own split logic. > FileInputFormat in MRv2 is a bit more configurable about input split > calculation & will split up large files. but otherwise, the partitioning is > being done more by the default values of the executing engine, rather than > any config data from the filesystem about what its "block size" is, > NativeAzureFS does a better job; maybe that could be factored out to > hadoop-common and reused? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364725#comment-16364725 ] Aaron Fabbri commented on HADOOP-13761: --- Finishing up the fail-on-read after open() retry stuff.. Should be posting a patch today. > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761.001.patch, HADOOP-13761.002.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14903) Add json-smart explicitly to pom.xml
[ https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reopened HADOOP-14903: --- > Add json-smart explicitly to pom.xml > > > Key: HADOOP-14903 > URL: https://issues.apache.org/jira/browse/HADOOP-14903 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14903-003-branch-2.patch, HADOOP-14903.001.patch, > HADOOP-14903.002.patch, HADOOP-14903.003.patch > > > With the library update in HADOOP-14799, maven knows how to pull in > net.minidev:json-smart for tests, but not for packaging. This needs to be > added to the main project pom in order to avoid this warning: > {noformat} > [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no > dependency information available > {noformat} > This is pulled in from a few places: > {noformat} > [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile > [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile > [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile > [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile > [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile > [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile > [INFO] | |+- > com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile > [INFO] | |\- net.minidev:json-smart:jar:2.3:compile > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364558#comment-16364558 ] Xiao Chen commented on HADOOP-12897: Reverted from trunk and branch-3.0. Seems branch-3.0.1 was not checked in so no action for 3.0.1 was taken. > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-12897: --- Fix Version/s: (was: 3.0.1) (was: 3.1.0) > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364568#comment-16364568 ] Ajay Kumar commented on HADOOP-12897: - [~xiaochen], thanks for reverting. Will check failed tests. > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364609#comment-16364609 ] Steve Loughran commented on HADOOP-14927: - +1, with that checkstyle (unused import) done before the commit thanks > ITestS3GuardTool failures in testDestroyNoBucket() > -- > > Key: HADOOP-14927 > URL: https://issues.apache.org/jira/browse/HADOOP-14927 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1, 3.0.0-alpha3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Minor > Attachments: HADOOP-14927.001.patch > > > Hit this when testing for the Hadoop 3.0.0-beta1 RC0. > {noformat} > hadoop-3.0.0-beta1-src/hadoop-tools/hadoop-aws$ mvn clean verify > -Dit.test="ITestS3GuardTool*" -Dtest=none -Ds3guard -Ddynamo > ... > Failed tests: > > ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228 > Expected an exception, got 0 > ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228 > Expected an exception, got 0 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15208) DistCp to offer option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15208: Attachment: HADOOP-15208-003.patch > DistCp to offer option to save src/dest filesets as alternative to delete() > --- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch, HADOOP-15208-003.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364600#comment-16364600 ] Hudson commented on HADOOP-12897: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13656 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13656/]) Revert "HADOOP-12897. KerberosAuthenticator.authenticate to include URL (xiao: rev 1f20f432d2472f92797ea01711ca4cc97e7b2b23) * (edit) hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java * (edit) hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[ https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aki Tanaka updated HADOOP-15206: Attachment: HADOOP-15206.006.patch > BZip2 drops and duplicates records when input split size is small > - > > Key: HADOOP-15206 > URL: https://issues.apache.org/jira/browse/HADOOP-15206 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.3, 3.0.0 >Reporter: Aki Tanaka >Priority: Major > Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch, > HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch, > HADOOP-15206.005.patch, HADOOP-15206.006.patch > > > BZip2 can drop and duplicate record when input split file is small. I > confirmed that this issue happens when the input split size is between 1byte > and 4bytes. > I am seeing the following 2 problem behaviors. > > 1. Drop record: > BZip2 skips the first record in the input file when the input split size is > small > > Set the split size to 3 and tested to load 100 records (0, 1, 2..99) > {code:java} > 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(317)) - > splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3 > count=99{code} > > The input format read only 99 records but not 100 records > > 2. Duplicate Record: > 2 input splits has same BZip2 records when the input split size is small > > Set the split size to 1 and tested to load 100 records (0, 1, 2..99) > > {code:java} > 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file > /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1 > count=99 > 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat > (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4 > at position 8 > {code} > > I experienced this error when I execute Spark (SparkSQL) job under the > following conditions: > * The file size of the input files are small (around 1KB) > * Hadoop cluster has many slave nodes (able to launch many executor tasks) > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[ https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14943: Status: Open (was: Patch Available) > Add common getFileBlockLocations() emulation for object stores, including S3A > - > > Key: HADOOP-14943 > URL: https://issues.apache.org/jira/browse/HADOOP-14943 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch, > HADOOP-14943-002.patch, HADOOP-14943-003.patch > > > It looks suspiciously like S3A isn't providing the partitioning data needed > in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a > file by the blocksize. This will stop tools using the MRv1 APIS doing the > partitioning properly if the input format isn't doing it own split logic. > FileInputFormat in MRv2 is a bit more configurable about input split > calculation & will split up large files. but otherwise, the partitioning is > being done more by the default values of the executing engine, rather than > any config data from the filesystem about what its "block size" is, > NativeAzureFS does a better job; maybe that could be factored out to > hadoop-common and reused? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12817) Enable TLS v1.1 and 1.2
[ https://issues.apache.org/jira/browse/HADOOP-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364632#comment-16364632 ] Brahma Reddy Battula commented on HADOOP-12817: --- Pushed to branch-2.8. There were minor conflicts,which are resolved(i.e changes.txt is not present ..). thanks [~jojochuang] > Enable TLS v1.1 and 1.2 > --- > > Key: HADOOP-12817 > URL: https://issues.apache.org/jira/browse/HADOOP-12817 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha1, 2.8.4 > > Attachments: HADOOP-12817.001.patch, HADOOP-12817.002.patch > > > Java 7 supports TLSv1.1 and TLSv1.2, which are more secure than TLSv1 (which > was all that was supported in Java 6), so we should add those to the default > list for {{hadoop.ssl.enabled.protocols}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[ https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364640#comment-16364640 ] genericqa commented on HADOOP-14903: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 28m 45s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 47s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14903 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910613/HADOOP-14903-003-branch-2.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux 68a15fd3a716 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1f20f43 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14121/testReport/ | | Max. process+thread count | 324 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14121/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Add json-smart explicitly to pom.xml > > > Key: HADOOP-14903 > URL: https://issues.apache.org/jira/browse/HADOOP-14903 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1 > > Attachments:
[jira] [Commented] (HADOOP-14734) add option to tag DDB table(s) created
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364660#comment-16364660 ] Steve Loughran commented on HADOOP-14734: - you've changed the line, it's picked up on an existing issue. The existing method shouldn't have done it. Change the param name to something else to make this go away, e.g "config" * revert field dynamoDBDocumentClient to dynamoDB; reduces change elsewhere, cost of merging other patches, etc. * I don't see where tagTable is being called. tests: * check the ordering of imports. * test needs to skip when * we are going to have to move off the TestMetadataStore altogether (see other JIRAs). Is this going to make that harder? Or, if we already have moved to an ITest only, make this test easier as a single method > add option to tag DDB table(s) created > -- > > Key: HADOOP-14734 > URL: https://issues.apache.org/jira/browse/HADOOP-14734 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Abraham Fine >Priority: Minor > Attachments: HADOOP-14734-001.patch, HADOOP-14734-002.patch > > > Many organisations have a "no untagged" resource policy; s3guard runs into > this when a table is created untagged. If there's a strict "delete untagged > resources" policy, the tables will go without warning. > Proposed: we add an option which can be used to declare the tags for a table > when created, use it in creation. No need to worry about updating/viewing > tags, as the AWS console can do that -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[ https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364663#comment-16364663 ] Steve Loughran commented on HADOOP-12897: - I'd seen the failures, but not put it down to this change. Looks like the failures are due to different exceptions being raised & their expectations broken. Interesting they didn't surface here though: why didn't this patch see them? > KerberosAuthenticator.authenticate to include URL on IO failures > > > Key: HADOOP-12897 > URL: https://issues.apache.org/jira/browse/HADOOP-12897 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Minor > Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, > HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch > > > If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you > get a stack trace, but without the URL it is trying to talk to. > That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} > handler —which can't be called here as its not in the {{hadoop-auth}} module -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org