[jira] [Created] (HADOOP-15112) create-release didn't sign artifacts
Andrew Wang created HADOOP-15112: Summary: create-release didn't sign artifacts Key: HADOOP-15112 URL: https://issues.apache.org/jira/browse/HADOOP-15112 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang While building the 3.0.0 RC1, I had to re-invoke Maven because the create-release script didn't deploy signatures to Nexus. Looking at the repo (and my artifacts), it seems like "sign" didn't run properly. I lost my create-release output, but I noticed that it will log and continue rather than abort in some error conditions. This might have caused my lack of signatures. IMO it'd be better to explicitly fail in these situations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
Andrew Wang created HADOOP-15058: Summary: create-release site build outputs dummy shaded jars due to skipShade Key: HADOOP-15058 URL: https://issues.apache.org/jira/browse/HADOOP-15058 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Blocker -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15037) Add site release notes for OrgQueue and resource types
Andrew Wang created HADOOP-15037: Summary: Add site release notes for OrgQueue and resource types Key: HADOOP-15037 URL: https://issues.apache.org/jira/browse/HADOOP-15037 Project: Hadoop Common Issue Type: Improvement Reporter: Andrew Wang Assignee: Andrew Wang Let's add some small blurbs and doc links to the site release notes for these features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15018) Update JAVA_HOME in create-release for Xenial Dockerfile
Andrew Wang created HADOOP-15018: Summary: Update JAVA_HOME in create-release for Xenial Dockerfile Key: HADOOP-15018 URL: https://issues.apache.org/jira/browse/HADOOP-15018 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Blocker create-release expects the Oracle JDK when setting JAVA_HOME. HADOOP-14816 no longer includes the Oracle JDK, so we need to update this to point to OpenJDK instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14555) document how to run wasb tests in azure docs site/testing.md
[
https://issues.apache.org/jira/browse/HADOOP-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-14555.
--
Resolution: Duplicate
Fix Version/s: (was: 3.0.0)
> document how to run wasb tests in azure docs site/testing.md
>
>
> Key: HADOOP-14555
> URL: https://issues.apache.org/jira/browse/HADOOP-14555
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> There's no single (current) documentation on running the azure tests
> * There's some in site/index.md, but iit looks potentially out of date
> (refers to an older azure SDK version)
> * There's a file
> {{src/test/org/apache/hadoop/fs/azure/RunningLiveWasbTests.txt}} which
> refers to a nonexistent doc {{hadoop-tools/hadoop-azure/README.txt }} for
> instructions.
> Proposed:
> # move testing docs out of main azure doc page, with link from there.
> # bring up to date with SDK, move of tests to ITests.
> # purge all other references, including bits of test javadocs which are no
> longer correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14555) document how to run wasb tests in azure docs site/testing.md
[
https://issues.apache.org/jira/browse/HADOOP-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-14555:
--
> document how to run wasb tests in azure docs site/testing.md
>
>
> Key: HADOOP-14555
> URL: https://issues.apache.org/jira/browse/HADOOP-14555
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> There's no single (current) documentation on running the azure tests
> * There's some in site/index.md, but iit looks potentially out of date
> (refers to an older azure SDK version)
> * There's a file
> {{src/test/org/apache/hadoop/fs/azure/RunningLiveWasbTests.txt}} which
> refers to a nonexistent doc {{hadoop-tools/hadoop-azure/README.txt }} for
> instructions.
> Proposed:
> # move testing docs out of main azure doc page, with link from there.
> # bring up to date with SDK, move of tests to ITests.
> # purge all other references, including bits of test javadocs which are no
> longer correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13917) Ensure yetus personality runs the integration tests for the shaded client
[ https://issues.apache.org/jira/browse/HADOOP-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-13917: -- > Ensure yetus personality runs the integration tests for the shaded client > - > > Key: HADOOP-13917 > URL: https://issues.apache.org/jira/browse/HADOOP-13917 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, test >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13917.WIP.0.patch, HADOOP-14771.02.patch > > > Either QBT or a different jenkins job should run our integration tests, > specifically the ones added for the shaded client. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13917) Ensure yetus personality runs the integration tests for the shaded client
[ https://issues.apache.org/jira/browse/HADOOP-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13917. -- Resolution: Delivered > Ensure yetus personality runs the integration tests for the shaded client > - > > Key: HADOOP-13917 > URL: https://issues.apache.org/jira/browse/HADOOP-13917 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, test >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13917.WIP.0.patch, HADOOP-14771.02.patch > > > Either QBT or a different jenkins job should run our integration tests, > specifically the ones added for the shaded client. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14545) Uninitialized S3A instance NPEs on toString()
[
https://issues.apache.org/jira/browse/HADOOP-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-14545.
--
Resolution: Duplicate
> Uninitialized S3A instance NPEs on toString()
> -
>
> Key: HADOOP-14545
> URL: https://issues.apache.org/jira/browse/HADOOP-14545
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.0.0-beta1
>
>
> You can't log an uninited S3AFileSystem instance without getting a stack trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getDefaultBlockSize(S3AFileSystem.java:2131)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.toString(S3AFileSystem.java:2148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14545) Uninitialized S3A instance NPEs on toString()
[
https://issues.apache.org/jira/browse/HADOOP-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-14545:
--
> Uninitialized S3A instance NPEs on toString()
> -
>
> Key: HADOOP-14545
> URL: https://issues.apache.org/jira/browse/HADOOP-14545
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.0.0-beta1
>
>
> You can't log an uninited S3AFileSystem instance without getting a stack trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getDefaultBlockSize(S3AFileSystem.java:2131)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.toString(S3AFileSystem.java:2148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14834) Remove original S3A output stream
[ https://issues.apache.org/jira/browse/HADOOP-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-14834. -- Resolution: Duplicate > Remove original S3A output stream > - > > Key: HADOOP-14834 > URL: https://issues.apache.org/jira/browse/HADOOP-14834 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.0.0-beta1 > > > The S3A Block output stream is working well and much better than the original > stream in terms of: scale, performance, instrumentation, robustness > Proposed: switch this to be the default, as a precursor to removing it later > HADOOP-14746 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14834) Remove original S3A output stream
[ https://issues.apache.org/jira/browse/HADOOP-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-14834: -- > Remove original S3A output stream > - > > Key: HADOOP-14834 > URL: https://issues.apache.org/jira/browse/HADOOP-14834 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.0.0-beta1 > > > The S3A Block output stream is working well and much better than the original > stream in terms of: scale, performance, instrumentation, robustness > Proposed: switch this to be the default, as a precursor to removing it later > HADOOP-14746 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14879) Build failure due to failing hadoop-client-check-invariants for hadoop-client-runtime.jar
[
https://issues.apache.org/jira/browse/HADOOP-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-14879.
--
Resolution: Done
> Build failure due to failing hadoop-client-check-invariants for
> hadoop-client-runtime.jar
> -
>
> Key: HADOOP-14879
> URL: https://issues.apache.org/jira/browse/HADOOP-14879
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.1.0
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Blocker
> Fix For: 3.0.0-beta1
>
>
> {noformat}
> [ERROR] Found artifact with unexpected contents:
> '/.../hadoop-client-modules/hadoop-client-runtime/target/hadoop-client-runtime-3.1.0-SNAPSHOT.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
> javax/
> javax/inject/
> javax/inject/Inject.class
> javax/inject/Named.class
> javax/inject/Provider.class
> javax/inject/Qualifier.class
> javax/inject/Scope.class
> javax/inject/Singleton.class
> jersey/
> jersey/repackaged/
> jersey/repackaged/org/
> jersey/repackaged/org/objectweb/
> jersey/repackaged/org/objectweb/asm/
> jersey/repackaged/org/objectweb/asm/AnnotationVisitor.class
> jersey/repackaged/org/objectweb/asm/AnnotationWriter.class
> jersey/repackaged/org/objectweb/asm/Attribute.class
> jersey/repackaged/org/objectweb/asm/ByteVector.class
> jersey/repackaged/org/objectweb/asm/ClassReader.class
> jersey/repackaged/org/objectweb/asm/ClassVisitor.class
> jersey/repackaged/org/objectweb/asm/ClassWriter.class
> jersey/repackaged/org/objectweb/asm/Context.class
> jersey/repackaged/org/objectweb/asm/Edge.class
> jersey/repackaged/org/objectweb/asm/FieldVisitor.class
> jersey/repackaged/org/objectweb/asm/FieldWriter.class
> jersey/repackaged/org/objectweb/asm/Frame.class
> jersey/repackaged/org/objectweb/asm/Handle.class
> jersey/repackaged/org/objectweb/asm/Handler.class
> jersey/repackaged/org/objectweb/asm/Item.class
> jersey/repackaged/org/objectweb/asm/Label.class
> jersey/repackaged/org/objectweb/asm/MethodVisitor.class
> jersey/repackaged/org/objectweb/asm/MethodWriter.class
> jersey/repackaged/org/objectweb/asm/Opcodes.class
> jersey/repackaged/org/objectweb/asm/Type.class
> jersey/repackaged/org/objectweb/asm/TypePath.class
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14879) Build failure due to failing hadoop-client-check-invariants for hadoop-client-runtime.jar
[
https://issues.apache.org/jira/browse/HADOOP-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-14879:
--
> Build failure due to failing hadoop-client-check-invariants for
> hadoop-client-runtime.jar
> -
>
> Key: HADOOP-14879
> URL: https://issues.apache.org/jira/browse/HADOOP-14879
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.1.0
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Blocker
> Fix For: 3.0.0-beta1
>
>
> {noformat}
> [ERROR] Found artifact with unexpected contents:
> '/.../hadoop-client-modules/hadoop-client-runtime/target/hadoop-client-runtime-3.1.0-SNAPSHOT.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
> javax/
> javax/inject/
> javax/inject/Inject.class
> javax/inject/Named.class
> javax/inject/Provider.class
> javax/inject/Qualifier.class
> javax/inject/Scope.class
> javax/inject/Singleton.class
> jersey/
> jersey/repackaged/
> jersey/repackaged/org/
> jersey/repackaged/org/objectweb/
> jersey/repackaged/org/objectweb/asm/
> jersey/repackaged/org/objectweb/asm/AnnotationVisitor.class
> jersey/repackaged/org/objectweb/asm/AnnotationWriter.class
> jersey/repackaged/org/objectweb/asm/Attribute.class
> jersey/repackaged/org/objectweb/asm/ByteVector.class
> jersey/repackaged/org/objectweb/asm/ClassReader.class
> jersey/repackaged/org/objectweb/asm/ClassVisitor.class
> jersey/repackaged/org/objectweb/asm/ClassWriter.class
> jersey/repackaged/org/objectweb/asm/Context.class
> jersey/repackaged/org/objectweb/asm/Edge.class
> jersey/repackaged/org/objectweb/asm/FieldVisitor.class
> jersey/repackaged/org/objectweb/asm/FieldWriter.class
> jersey/repackaged/org/objectweb/asm/Frame.class
> jersey/repackaged/org/objectweb/asm/Handle.class
> jersey/repackaged/org/objectweb/asm/Handler.class
> jersey/repackaged/org/objectweb/asm/Item.class
> jersey/repackaged/org/objectweb/asm/Label.class
> jersey/repackaged/org/objectweb/asm/MethodVisitor.class
> jersey/repackaged/org/objectweb/asm/MethodWriter.class
> jersey/repackaged/org/objectweb/asm/Opcodes.class
> jersey/repackaged/org/objectweb/asm/Type.class
> jersey/repackaged/org/objectweb/asm/TypePath.class
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11656) Classpath isolation for downstream clients
[ https://issues.apache.org/jira/browse/HADOOP-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-11656. -- Resolution: Done Hadoop Flags: (was: Incompatible change) Fix Version/s: 3.0.0-beta1 I'm resolving this as Done since all subtasks have been completed. There are still a few follow-ons being tracked for 3.0.0 GA. Many thanks to Sean for driving this forward! Great work! > Classpath isolation for downstream clients > -- > > Key: HADOOP-11656 > URL: https://issues.apache.org/jira/browse/HADOOP-11656 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Blocker > Labels: classloading, classpath, dependencies, scripts, shell > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-11656_proposal.md > > > Currently, Hadoop exposes downstream clients to a variety of third party > libraries. As our code base grows and matures we increase the set of > libraries we rely on. At the same time, as our user base grows we increase > the likelihood that some downstream project will run into a conflict while > attempting to use a different version of some library we depend on. This has > already happened with i.e. Guava several times for HBase, Accumulo, and Spark > (and I'm sure others). > While YARN-286 and MAPREDUCE-1700 provided an initial effort, they default to > off and they don't do anything to help dependency conflicts on the driver > side or for folks talking to HDFS directly. This should serve as an umbrella > for changes needed to do things thoroughly on the next major version. > We should ensure that downstream clients > 1) can depend on a client artifact for each of HDFS, YARN, and MapReduce that > doesn't pull in any third party dependencies > 2) only see our public API classes (or as close to this as feasible) when > executing user provided code, whether client side in a launcher/driver or on > the cluster in a container or within MR. > This provides us with a double benefit: users get less grief when they want > to run substantially ahead or behind the versions we need and the project is > freer to change our own dependency versions because they'll no longer be in > our compatibility promises. > Project specific task jiras to follow after I get some justifying use cases > written in the comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14655) Update httpcore version to 4.4.6
[ https://issues.apache.org/jira/browse/HADOOP-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-14655: -- Reverted this JIRA from trunk and branch-3.0 per Marton's instructions. > Update httpcore version to 4.4.6 > > > Key: HADOOP-14655 > URL: https://issues.apache.org/jira/browse/HADOOP-14655 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Ray Chiang > Attachments: HADOOP-14655.001.patch > > > Update the dependency > org.apache.httpcomponents:httpcore:4.4.4 > to the latest (4.4.6). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14655) Update httpcore version to 4.4.6
[ https://issues.apache.org/jira/browse/HADOOP-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-14655. -- Resolution: Fixed While it's in the branch, let's leave the JIRA resolved for release notes purposes. > Update httpcore version to 4.4.6 > > > Key: HADOOP-14655 > URL: https://issues.apache.org/jira/browse/HADOOP-14655 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Ray Chiang > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14655.001.patch > > > Update the dependency > org.apache.httpcomponents:httpcore:4.4.4 > to the latest (4.4.6). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14848) Switch from JDiff to japicmp
Andrew Wang created HADOOP-14848: Summary: Switch from JDiff to japicmp Key: HADOOP-14848 URL: https://issues.apache.org/jira/browse/HADOOP-14848 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha4 Reporter: Andrew Wang JDiff is old and not maintained. It complicates our build by requiring xerces, and also a lot of Maven logic to custom patch it and stitch it up. japicmp was proposed as a more up-to-date tool that also has a Maven plugin. It's also ALv2 which is a nice bonus. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13998) Merge initial S3guard release into trunk
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13998. -- Resolution: Done Re-resolving per above. > Merge initial S3guard release into trunk > > > Key: HADOOP-13998 > URL: https://issues.apache.org/jira/browse/HADOOP-13998 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13998-001.patch, HADOOP-13998-002.patch, > HADOOP-13998-003.patch, HADOOP-13998-004.patch, HADOOP-13998-005.patch > > > JIRA to link in all the things we think are needed for a preview/merge into > trunk -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13998) Merge initial S3guard release into trunk
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-13998: -- Re-opening to resolve as "Complete" or something, since this code change was attributed to the parent JIRA HADOOP-13345 in the commit message. > Merge initial S3guard release into trunk > > > Key: HADOOP-13998 > URL: https://issues.apache.org/jira/browse/HADOOP-13998 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13998-001.patch, HADOOP-13998-002.patch, > HADOOP-13998-003.patch, HADOOP-13998-004.patch, HADOOP-13998-005.patch > > > JIRA to link in all the things we think are needed for a preview/merge into > trunk -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-14674:
--
I don't see this showing up in trunk, did it actually get committed?
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14805) Upgrade to zstd 1.3.1
Andrew Wang created HADOOP-14805: Summary: Upgrade to zstd 1.3.1 Key: HADOOP-14805 URL: https://issues.apache.org/jira/browse/HADOOP-14805 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha2, 2.9.0 Reporter: Andrew Wang -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14731) Update gitignore to exclude output of site build
Andrew Wang created HADOOP-14731: Summary: Update gitignore to exclude output of site build Key: HADOOP-14731 URL: https://issues.apache.org/jira/browse/HADOOP-14731 Project: Hadoop Common Issue Type: Improvement Components: build, site Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang Assignee: Andrew Wang Site build generates a bunch of files that aren't caught by gitignore, let's update. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14501) aalto-xml cannot handle some odd XML features
Andrew Wang created HADOOP-14501:
Summary: aalto-xml cannot handle some odd XML features
Key: HADOOP-14501
URL: https://issues.apache.org/jira/browse/HADOOP-14501
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.9.0, 3.0.0-alpha4
Reporter: Andrew Wang
Priority: Blocker
[~hgadre] tried testing solr with a Hadoop 3 client. He saw various test case
failures due to what look like functionality gaps in the new aalto-xml stax
implementation pulled in by HADOOP-14216:
{noformat}
[junit4]> Throwable #1: com.fasterxml.aalto.WFCException: Illegal XML
character ('ΓΌ' (code 252))
[junit4]> Caused by: com.fasterxml.aalto.WFCException: General entity
reference () encountered in entity expanding mode: operation not (yet)
implemented
...
[junit4]> Throwable #1: org.apache.solr.common.SolrException: General
entity reference () encountered in entity expanding mode: operation not
(yet) implemented
{noformat}
These were from the following test case executions:
{noformat}
NOTE: reproduce with: ant test -Dtestcase=DocumentAnalysisRequestHandlerTest
-Dtests.method=testCharsetOutsideDocument -Dtests.seed=2F739D88D9C723CA
-Dtests.slow=true -Dtests.locale=und -Dtests.timezone=Atlantic/Faeroe
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII
NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest
-Dtests.method=testXMLDiffWithExternalEntity -Dtests.seed=2F739D88D9C723CA
-Dtests.slow=true -Dtests.locale=en-US -Dtests.timezone=US/Aleutian
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII
NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
-Dtests.method=testExternalEntities -Dtests.seed=2F739D88D9C723CA
-Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Barbados
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII
NOTE: reproduce with: ant test -Dtestcase=XmlUpdateRequestHandlerTest
-Dtests.method=testNamedEntity -Dtests.seed=2F739D88D9C723CA -Dtests.slow=true
-Dtests.locale=hr -Dtests.timezone=America/Barbados -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14399) Configuration does not correctly XInclude absolute file URIs
Andrew Wang created HADOOP-14399:
Summary: Configuration does not correctly XInclude absolute file
URIs
Key: HADOOP-14399
URL: https://issues.apache.org/jira/browse/HADOOP-14399
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.9.0, 3.0.0-alpha3
Reporter: Andrew Wang
Priority: Blocker
[Reported
by|https://issues.apache.org/jira/browse/HADOOP-14216?focusedCommentId=15967816=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15967816]
[~ste...@apache.org] on HADOOP-14216, filing this JIRA on his behalf:
{quote}
Just tracked this down as the likely cause of my S3A test failures. This is
pulling in core-site.xml, which then xincludes auth-keys.xml, which finally
references an absolute path, file://home/stevel/(secret)/aws-keys.xml. This is
failing for me even with the latest patch in. Either transient XIncludes aren't
being picked up or
Note also I think the error could be improved. 1. It's in the included file
where the problem appears to lie and 2. we should really know the missing
entry. Perhaps a wiki link too: I had to read the XInclude spec to work out
what was going on here before I could go back to finding the cause
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14216) Improve Configuration XML Parsing Performance
[ https://issues.apache.org/jira/browse/HADOOP-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-14216. -- Resolution: Fixed I'm going to re-resolving this so the release notes are consistent. If we revert this JIRA, then it's appropriate to re-open. I'll file a follow-on issue for tracking. > Improve Configuration XML Parsing Performance > - > > Key: HADOOP-14216 > URL: https://issues.apache.org/jira/browse/HADOOP-14216 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-14216.1.patch, HADOOP-14216.2-branch-2.patch, > HADOOP-14216.2.patch, HADOOP-14216.addendum.1.patch > > > JIRA is to improve XML parsing performance through reuse and a change in XML > parser (STAX) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14386) Make trunk work with Guava 11.0.2 again
Andrew Wang created HADOOP-14386: Summary: Make trunk work with Guava 11.0.2 again Key: HADOOP-14386 URL: https://issues.apache.org/jira/browse/HADOOP-14386 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang As an alternative to reverting or shading HADOOP-10101 (the upgrade of Guava from 11.0.2 to 21.0), HADOOP-14380 makes the Guava version configurable. However, it still doesn't compile with Guava 11.0.2, since HADOOP-10101 chose to use the moved Guava classes rather than replacing them with alternatives. This JIRA aims to make Hadoop compatible with Guava 11.0.2 as well as 21.0 by replacing usage of these moved Guava classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14382) Remove usages of MoreObjects.toStringHelper
Andrew Wang created HADOOP-14382: Summary: Remove usages of MoreObjects.toStringHelper Key: HADOOP-14382 URL: https://issues.apache.org/jira/browse/HADOOP-14382 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0-alpha2 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor MoreObjects.toStringHelper is a source of incompatibility across Guava versions. Let's move off of this to a native Java 8 API. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-10604) CryptoFileSystem decorator using xAttrs and KeyProvider
[ https://issues.apache.org/jira/browse/HADOOP-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-10604. -- Resolution: Incomplete I think we decided not to pursue this effort further, and focused instead on HDFS transparent encryption. Resolving. > CryptoFileSystem decorator using xAttrs and KeyProvider > --- > > Key: HADOOP-10604 > URL: https://issues.apache.org/jira/browse/HADOOP-10604 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Alejandro Abdelnur >Assignee: Yi Liu > Attachments: HADOOP-10604.1.patch, HADOOP-10604.patch > > > A FileSystem implementation that wraps an existing filesystem and provides > encryption. It will require the underlying filesystem to support xAttrs. It > will use the KeyProvider API to retrieve encryption keys. > This is mostly the work in the patch HADOOP-10150 minus the crypto streams -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14293) Initialize FakeTimer with a less trivial value.
Andrew Wang created HADOOP-14293: Summary: Initialize FakeTimer with a less trivial value. Key: HADOOP-14293 URL: https://issues.apache.org/jira/browse/HADOOP-14293 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang Assignee: Andrew Wang HADOOP-14276 broke TestFsDatasetImpl#testLoadingDfsUsedForVolumes which uses a FakeTimer. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14284) Shade Guava everywhere
Andrew Wang created HADOOP-14284:
Summary: Shade Guava everywhere
Key: HADOOP-14284
URL: https://issues.apache.org/jira/browse/HADOOP-14284
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0-alpha3
Reporter: Andrew Wang
Priority: Blocker
HADOOP-10101 upgraded the guava version for 3.x to 21.
Guava is broadly used by Java projects that consume our artifacts.
Unfortunately, these projects also consume our private artifacts like
{{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced by
HADOOP-11804, currently only available in 3.0.0-alpha2.
We should shade Guava everywhere to proactively avoid breaking downstreams.
This isn't a requirement for all dependency upgrades, but it's necessary for
known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14280) Fix compilation of TestKafkaMetrics
Andrew Wang created HADOOP-14280: Summary: Fix compilation of TestKafkaMetrics Key: HADOOP-14280 URL: https://issues.apache.org/jira/browse/HADOOP-14280 Project: Hadoop Common Issue Type: Improvement Components: tools Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang Assignee: Andrew Wang The upgrade to Guava 21 at HADOOP-10101 missed updating this Kafka test. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14257) hadoop-auth and hadoop-annotations jars are in lib directory
Andrew Wang created HADOOP-14257:
Summary: hadoop-auth and hadoop-annotations jars are in lib
directory
Key: HADOOP-14257
URL: https://issues.apache.org/jira/browse/HADOOP-14257
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.0.0-alpha2, 2.8.0
Reporter: Andrew Wang
Poking around in the 3.0.0-alpha2 tarball, noticed that the auth and
annotations JARs seem to be in the wrong place (lib dir):
{noformat}
./share/hadoop/common/lib/hadoop-annotations-3.0.0-alpha2.jar
./share/hadoop/common/lib/hadoop-auth-3.0.0-alpha2.jar
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-9357) Fallback to default authority if not specified in FileContext
[
https://issues.apache.org/jira/browse/HADOOP-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-9357.
-
Resolution: Invalid
I'm going to just resolve this one, unlikely we're going to make any
FileContext changes at this point.
> Fallback to default authority if not specified in FileContext
> -
>
> Key: HADOOP-9357
> URL: https://issues.apache.org/jira/browse/HADOOP-9357
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Minor
> Attachments: hadoop-9357-1.patch, hadoop-9357-2.patch,
> hadoop-9357-3.patch, hadoop-9357-testfixup.patch
>
>
> Currently, FileContext adheres rather strictly to RFC2396 when it comes to
> parsing absolute URIs (URIs with a scheme). If a user asks for a URI like
> "hdfs:///tmp", FileContext will error while FileSystem will add the authority
> of the default FS (e.g. turn it into "hdfs://defaultNN:port/tmp").
> This is technically correct, but FileSystem's behavior is nicer for users and
> okay based on 5.2.3 in the RFC, so lets do it in FileContext too:
> {noformat}
> For backwards
> compatibility, an implementation may work around such references
> by removing the scheme if it matches that of the base URI and the
> scheme is known to always use the syntax. The parser
> can then continue with the steps below for the remainder of the
> reference components. Validating parsers should mark such a
> misformed relative reference as an error.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-10856) HarFileSystem and HarFs support for HDFS encryption
[ https://issues.apache.org/jira/browse/HADOOP-10856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-10856. -- Resolution: Won't Fix I agree with Tucu's assessment above, we don't need to do finer-grained encryption inside a HAR. It's a file, we can treat it like a file. Resolving this old JIRA. > HarFileSystem and HarFs support for HDFS encryption > --- > > Key: HADOOP-10856 > URL: https://issues.apache.org/jira/browse/HADOOP-10856 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Andrew Wang >Assignee: Andrew Wang > > We need to examine support for Har with HDFS encryption. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14045) Aliyun OSS documentation missing from website
Andrew Wang created HADOOP-14045: Summary: Aliyun OSS documentation missing from website Key: HADOOP-14045 URL: https://issues.apache.org/jira/browse/HADOOP-14045 Project: Hadoop Common Issue Type: Bug Components: documentation, fs/oss Affects Versions: 3.0.0-alpha2 Reporter: Andrew Wang I'm looking at the alpha2 website, and can't find a link to the Aliyun OSS documentation. Under the "Hadoop Compatible File Systems" header there are links to S3, Azure blob, ADLS, and Swift, but not Aliyun OSS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14024) KMS JMX endpoint throws ClassNotFoundException
Andrew Wang created HADOOP-14024:
Summary: KMS JMX endpoint throws ClassNotFoundException
Key: HADOOP-14024
URL: https://issues.apache.org/jira/browse/HADOOP-14024
Project: Hadoop Common
Issue Type: Bug
Components: kms
Affects Versions: 2.8.0
Reporter: Andrew Wang
Throws like this:
{noformat}
root cause java.lang.ClassNotFoundException:
org.mortbay.jetty.servlet.Context
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698)
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:174)
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:588)
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129)
{noformat}
I tried out branch-2.6 and it seems to be okay, so something changed between
2.6.x and 2.8.x/branch-2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14014) Shading runs on mvn deploy
Andrew Wang created HADOOP-14014: Summary: Shading runs on mvn deploy Key: HADOOP-14014 URL: https://issues.apache.org/jira/browse/HADOOP-14014 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0-alpha2 Reporter: Andrew Wang I'm running "mvn deploy -DskipTests" and see that there is shading happening in the build output. This seems like a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14003) Make additional KMS tomcat settings configurable
Andrew Wang created HADOOP-14003:
Summary: Make additional KMS tomcat settings configurable
Key: HADOOP-14003
URL: https://issues.apache.org/jira/browse/HADOOP-14003
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 2.8.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Doing some Tomcat performance tuning on a loaded cluster, we found that
{{acceptCount}}, {{acceptorThreadCount}}, and {{protocol}} can be useful. Let's
make these configurable in the kms startup script.
Since the KMS is Jetty in 3.x, this is targeted at just branch-2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13996) Fix some release build issues
Andrew Wang created HADOOP-13996: Summary: Fix some release build issues Key: HADOOP-13996 URL: https://issues.apache.org/jira/browse/HADOOP-13996 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0-alpha2 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Blocker Found some build issues while doing some test runs with the create-release.sh script. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13978) Update project release notes for 3.0.0-alpha2
Andrew Wang created HADOOP-13978: Summary: Update project release notes for 3.0.0-alpha2 Key: HADOOP-13978 URL: https://issues.apache.org/jira/browse/HADOOP-13978 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha2 Reporter: Andrew Wang Assignee: Andrew Wang Let's update the website release notes for 3.0.0-alpha2's changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13136) shade protobuf in the hadoop-common jar
[ https://issues.apache.org/jira/browse/HADOOP-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13136. -- Resolution: Duplicate Fix Version/s: 3.0.0-alpha2 Resolving since HADOOP-11804 has been committed. > shade protobuf in the hadoop-common jar > --- > > Key: HADOOP-13136 > URL: https://issues.apache.org/jira/browse/HADOOP-13136 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Haohui Mai > Fix For: 3.0.0-alpha2 > > > While Protobuf has good wire compatibility, its implementation has been > changed from time to time. It might be a good idea to shade it in for better > compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13586) Hadoop 3.0 build broken on windows
[ https://issues.apache.org/jira/browse/HADOOP-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13586. -- Resolution: Cannot Reproduce I'm going to close this since Chris tested this in late October, and there haven't been any responses to requests for additional info. If we get additional info, we can reopen to track a fix. > Hadoop 3.0 build broken on windows > -- > > Key: HADOOP-13586 > URL: https://issues.apache.org/jira/browse/HADOOP-13586 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 > Environment: Windows Server >Reporter: Steve Loughran >Priority: Blocker > > Builds on windows fail, even before getting to the native bits > Looks like dev-support/bin/dist-copynativelibs isn't windows-ready -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13663) Index out of range in SysInfoWindows
[
https://issues.apache.org/jira/browse/HADOOP-13663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-13663.
--
Resolution: Fixed
Re-resolving so the status is "Fixed" rather than "Resolved".
> Index out of range in SysInfoWindows
>
>
> Key: HADOOP-13663
> URL: https://issues.apache.org/jira/browse/HADOOP-13663
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.3
> Environment: Windows
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13663.000.patch, HADOOP-13663.001.patch
>
>
> Sometimes, the {{NodeResourceMonitor}} tries to read the system utilization
> from winutils.exe and this return empty values. This triggers the following
> exception:
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1911)
> at
> org.apache.hadoop.util.SysInfoWindows.refreshIfNeeded(SysInfoWindows.java:158)
> at
> org.apache.hadoop.util.SysInfoWindows.getPhysicalMemorySize(SysInfoWindows.java:247)
> at
> org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.getPhysicalMemorySize(ResourceCalculatorPlugin.java:63)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl$MonitoringThread.run(NodeResourceMonitorImpl.java:139)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13663) Index out of range in SysInfoWindows
[
https://issues.apache.org/jira/browse/HADOOP-13663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-13663:
--
> Index out of range in SysInfoWindows
>
>
> Key: HADOOP-13663
> URL: https://issues.apache.org/jira/browse/HADOOP-13663
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.3
> Environment: Windows
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13663.000.patch, HADOOP-13663.001.patch
>
>
> Sometimes, the {{NodeResourceMonitor}} tries to read the system utilization
> from winutils.exe and this return empty values. This triggers the following
> exception:
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1911)
> at
> org.apache.hadoop.util.SysInfoWindows.refreshIfNeeded(SysInfoWindows.java:158)
> at
> org.apache.hadoop.util.SysInfoWindows.getPhysicalMemorySize(SysInfoWindows.java:247)
> at
> org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.getPhysicalMemorySize(ResourceCalculatorPlugin.java:63)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl$MonitoringThread.run(NodeResourceMonitorImpl.java:139)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13423) Run JDiff on trunk for Hadoop-Common and analyze results
[ https://issues.apache.org/jira/browse/HADOOP-13423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13423. -- Resolution: Duplicate I'd like to close this in favor of the JACC reports. I checked the JACC report for alpha1, and found it easier to use than jdiff. > Run JDiff on trunk for Hadoop-Common and analyze results > > > Key: HADOOP-13423 > URL: https://issues.apache.org/jira/browse/HADOOP-13423 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: 3.0.0-alpha1-hadoop-common-jdiff.zip, > 3.0.0-alpha1-jdiff.zip > > > We need to run JDiff and make sure the first 3.0.0 alpha release doesn't > include unnecessary API incompatible change. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13543) [Umbrella] Analyse 2.8.0 and 3.0.0-alpha1 jdiff reports and fix any issues
[ https://issues.apache.org/jira/browse/HADOOP-13543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13543. -- Resolution: Duplicate Been a month since my last comment. Seems like we can close this as dupe to the ACC work. Please reopen if you disagree. > [Umbrella] Analyse 2.8.0 and 3.0.0-alpha1 jdiff reports and fix any issues > -- > > Key: HADOOP-13543 > URL: https://issues.apache.org/jira/browse/HADOOP-13543 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > > Now that we have fixed JDiff report generation for 2.8.0 and above, we should > analyse them. > For the previous releases, I was applying the jdiff patches myself, and > analysed them offline. It's better to track them here now that the reports > are automatically getting generated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13803) Revert HADOOP-13081 add the ability to create multiple UGIs/subjects from one kerberos login
[ https://issues.apache.org/jira/browse/HADOOP-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13803. -- Resolution: Fixed Fix Version/s: 3.0.0-alpha2 > Revert HADOOP-13081 add the ability to create multiple UGIs/subjects from one > kerberos login > > > Key: HADOOP-13803 > URL: https://issues.apache.org/jira/browse/HADOOP-13803 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Chris Nauroth > Fix For: 3.0.0-alpha2 > > > HADOOP-13081 was released in 3.0.0-alpha1 and the revert wasn't tracked in a > separate JIRA. Filing this one to correct the changelog/release for alpha2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13803) Revert HADOOP-13081 add the ability to create multiple UGIs/subjects from one kerberos login
Andrew Wang created HADOOP-13803: Summary: Revert HADOOP-13081 add the ability to create multiple UGIs/subjects from one kerberos login Key: HADOOP-13803 URL: https://issues.apache.org/jira/browse/HADOOP-13803 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Assignee: Chris Nauroth HADOOP-13081 was released in 3.0.0-alpha1 and the revert wasn't tracked in a separate JIRA. Filing this one to correct the changelog/release for alpha2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13790) Make qbt script executable
Andrew Wang created HADOOP-13790:
Summary: Make qbt script executable
Key: HADOOP-13790
URL: https://issues.apache.org/jira/browse/HADOOP-13790
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0-alpha1, 2.8.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Trivial
Attachments: HADOOP-13790.001.patch
Trivial, the qbt script isn't executable, unlike the other scripts in
{{dev-support/bin}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-13717.
--
Resolution: Invalid
Fix Version/s: (was: 3.0.0-alpha2)
Reverted and reclosing, thanks again Anu for catching this.
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-13717:
--
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13784) Output javadoc inside the target directory
Andrew Wang created HADOOP-13784:
Summary: Output javadoc inside the target directory
Key: HADOOP-13784
URL: https://issues.apache.org/jira/browse/HADOOP-13784
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Andrew Wang
Assignee: Andrew Wang
HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
javadoc dirs being created outside target folders:
{noformat}
hadoop-common-project/hadoop-common/api/
hadoop-hdfs-project/hadoop-hdfs-client/api/
hadoop-hdfs-project/hadoop-hdfs/api/
{noformat}
Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13696) change hadoop-common dependency scope of jsch to provided.
[ https://issues.apache.org/jira/browse/HADOOP-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13696. -- Resolution: Won't Fix Fix Version/s: (was: 3.0.0-alpha2) I filed and linked HADOOP-13769 for splitting this out, thanks again all. > change hadoop-common dependency scope of jsch to provided. > -- > > Key: HADOOP-13696 > URL: https://issues.apache.org/jira/browse/HADOOP-13696 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Yuanbo Liu >Priority: Minor > Attachments: HADOOP-13696.001.patch > > > The dependency on jsch in Hadoop common is "compile", so it gets everywhere > downstream. Marking it as "provided" would mean that it would only be needed > by those programs which wanted the SFTP filesystem, and, if they wanted to > use a different jsch version, there'd be no maven problems -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13696) change hadoop-common dependency scope of jsch to provided.
[ https://issues.apache.org/jira/browse/HADOOP-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-13696: -- > change hadoop-common dependency scope of jsch to provided. > -- > > Key: HADOOP-13696 > URL: https://issues.apache.org/jira/browse/HADOOP-13696 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Yuanbo Liu >Priority: Minor > Attachments: HADOOP-13696.001.patch > > > The dependency on jsch in Hadoop common is "compile", so it gets everywhere > downstream. Marking it as "provided" would mean that it would only be needed > by those programs which wanted the SFTP filesystem, and, if they wanted to > use a different jsch version, there'd be no maven problems -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13759) Split SFTP FileSystem into its own artifact
Andrew Wang created HADOOP-13759: Summary: Split SFTP FileSystem into its own artifact Key: HADOOP-13759 URL: https://issues.apache.org/jira/browse/HADOOP-13759 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.3 Reporter: Andrew Wang As discussed on HADOOP-13696, if we split the SFTP FileSystem into its own artifact, we can save a jsch dependency in Hadoop Common. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-9280) HADOOP-7101 was never merged from 0.23.x to the 1.x branch
[ https://issues.apache.org/jira/browse/HADOOP-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-9280. - Resolution: Won't Fix Old JIRA for branch-1, resolving. > HADOOP-7101 was never merged from 0.23.x to the 1.x branch > -- > > Key: HADOOP-9280 > URL: https://issues.apache.org/jira/browse/HADOOP-9280 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 1.0.4 >Reporter: Claus Ibsen >Assignee: Suresh Srinivas >Priority: Critical > > See HADOOP-7101 > This code fix went into the 0.23 branch. > But was never merged into the 1.x branch, which causing problems for people > upgrading from 0.23 to 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-8190) Eclipse plugin fails to access remote cluster
[ https://issues.apache.org/jira/browse/HADOOP-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-8190. - Resolution: Not A Problem This JIRA is 4 years old, I'm going to resolve. Please reopen if you intend to work on it. > Eclipse plugin fails to access remote cluster > - > > Key: HADOOP-8190 > URL: https://issues.apache.org/jira/browse/HADOOP-8190 > Project: Hadoop Common > Issue Type: Bug > Components: contrib/eclipse-plugin >Affects Versions: 0.20.205.0 > Environment: Windows and Linux (all) >Reporter: Ambud Sharma >Priority: Critical > Labels: gsoc2012 > Original Estimate: 12h > Remaining Estimate: 12h > > Eclipse plugin fails to access remote file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13724) Fix a few typos in site markdown documents
Andrew Wang created HADOOP-13724: Summary: Fix a few typos in site markdown documents Key: HADOOP-13724 URL: https://issues.apache.org/jira/browse/HADOOP-13724 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 3.0.0-alpha1, 2.8.0 Reporter: Andrew Wang Assignee: Ding Fei Priority: Minor Attachments: HADOOP-13708-4.patch New JIRA for HADOOP-13708 since precommit bot is confused by the combination of PRs and patches. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
Andrew Wang created HADOOP-13717:
Summary: Shell scripts call hadoop_verify_logdir even when command
is not started as daemon
Key: HADOOP-13717
URL: https://issues.apache.org/jira/browse/HADOOP-13717
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0-alpha1
Reporter: Andrew Wang
Issue found when working with the HDFS balancer.
In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
"default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
specifies the log location isn't even used here, since the command is being
started in the foreground.
I think we can push the {{hadoop_verify_logdir}} call down into
{{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13705) Revert HADOOP-13534 Remove unused TrashPolicy#getInstance and initialize code
Andrew Wang created HADOOP-13705: Summary: Revert HADOOP-13534 Remove unused TrashPolicy#getInstance and initialize code Key: HADOOP-13705 URL: https://issues.apache.org/jira/browse/HADOOP-13705 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Assignee: Andrew Wang Per discussion on HADOOP-13700, I'd like to revert HADOOP-13534. It removes a deprecated API, but the 2.x line does not have a release with the new replacement API. This places a burden on downstream applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13699) Configuration does not substitute multiple references to the same var
[
https://issues.apache.org/jira/browse/HADOOP-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-13699.
--
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Committed to trunk, thanks for reviewing Xiao!
> Configuration does not substitute multiple references to the same var
> -
>
> Key: HADOOP-13699
> URL: https://issues.apache.org/jira/browse/HADOOP-13699
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13699.001.patch
>
>
> Config var loop detection was originally introduced by HADOOP-6871. Due to
> cycle detection changes in the trunk patch for HADOOP-11506, resolution for
> multiple references to the same variable no longer resolved, e.g.
> {noformat}
> somekey = "${otherkey} ${otherkey}"
> {noformat}
> This loop detection business is fragile, expensive, and not in branch-2, so
> let's reduce it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13699) Configuration does not substitute multiple references to the same var
Andrew Wang created HADOOP-13699:
Summary: Configuration does not substitute multiple references to
the same var
Key: HADOOP-13699
URL: https://issues.apache.org/jira/browse/HADOOP-13699
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 3.0.0-alpha1
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Critical
Config var loop detection was originally introduced by HADOOP-6871. Due to
cycle detection changes in the trunk patch for HADOOP-11506, resolution for
multiple references to the same variable no longer resolved, e.g.
{noformat}
somekey = "${otherkey} ${otherkey}"
{noformat}
This loop detection business is fragile, expensive, and not in branch-2, so
let's reduce it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11090) [Umbrella] Support Java 8 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-11090. -- Resolution: Fixed I think we're safe to resolve this JIRA. CDH blacklists a few versions, but certified with JDK8 (based on heavily modified 2.6): https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_jdk HDP seems similar (2.7 based): http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_Installing_HDP_AMB/content/_jdk_requirements.html We also bumped the required JDK version to JDK8 for 3.0.0-alpha1. If there are additional JDK8 issues, let's follow up with separate JIRAs. Thanks all. > [Umbrella] Support Java 8 in Hadoop > --- > > Key: HADOOP-11090 > URL: https://issues.apache.org/jira/browse/HADOOP-11090 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Mohammad Kamrul Islam >Assignee: Mohammad Kamrul Islam > > Java 8 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 8 is important for the Apache community. > > This JIRA is to track the issues/experiences encountered during Java 8 > migration. If you find a potential bug , please create a separate JIRA either > as a sub-task or linked into this JIRA. > If you find a Hadoop or JVM configuration tuning, you can create a JIRA as > well. Or you can add a comment here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13632) Daemonization does not check process liveness before renicing
Andrew Wang created HADOOP-13632:
Summary: Daemonization does not check process liveness before
renicing
Key: HADOOP-13632
URL: https://issues.apache.org/jira/browse/HADOOP-13632
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0-alpha1
Reporter: Andrew Wang
If you try to daemonize a process that is incorrectly configured, it will die
quite quickly. However, the daemonization function will still try to renice it
even if it's down, leading to something like this for my namenode:
{noformat}
-> % bin/hdfs --daemon start namenode
ERROR: Cannot set priority of namenode process 12036
{noformat}
It'd be more user-friendly instead of this renice error, we said that the
process couldn't be started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13615) Convert uses of AtomicLong for counter metrics to LongAdder
Andrew Wang created HADOOP-13615: Summary: Convert uses of AtomicLong for counter metrics to LongAdder Key: HADOOP-13615 URL: https://issues.apache.org/jira/browse/HADOOP-13615 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang LongAdder (available in JDK8) can provide much better performance than AtomicLong since it uses thread locals under the hood. We should consider switching over our uses of AtomicLong and friends over to LongAdder. If we want to target for JDK7, we can also pull in the implementation since it's pure Java (public domain): http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-7198) Hadoop defaults for web UI ports often fall smack in the middle of Linux ephemeral port range
[ https://issues.apache.org/jira/browse/HADOOP-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-7198. - Resolution: Duplicate Believe this is handled by HDFS-9427 and related JIRAs. Resolving; any remaining ports in the ephemeral range can be addressed in specific JIRAs. > Hadoop defaults for web UI ports often fall smack in the middle of Linux > ephemeral port range > - > > Key: HADOOP-7198 > URL: https://issues.apache.org/jira/browse/HADOOP-7198 > Project: Hadoop Common > Issue Type: Wish >Reporter: Philip Zeyliger >Priority: Trivial > > It turns out (see http://en.wikipedia.org/wiki/Ephemeral_port and > /proc/sys/net/ipv4/ip_local_port_range) that when you bind to port 0, Linux > chooses an ephemeral port. On my default-ridden Ubuntu Maverick box and on > CentOS 5.5, that range is 32768-61000. So, when HBase binds to 60030 or when > mapReduce binds to 50070, there's a small chance that you'll conflict with, > say, an FTP session, or with some other Hadoop daemon that's had a listening > address configured as :0. > I don't know that there's a practical resolution here, since changing the > defaults seems like an ill-fated effort, but if you have any ephemeral port > use, you can run into this. We've now run into it once. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13575) Website shows incorrect 25 January, 2016 as the release date for 2.7.3
[ https://issues.apache.org/jira/browse/HADOOP-13575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13575. -- Resolution: Done I noticed this independently and fixed already, should be good. > Website shows incorrect 25 January, 2016 as the release date for 2.7.3 > -- > > Key: HADOOP-13575 > URL: https://issues.apache.org/jira/browse/HADOOP-13575 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Jacek Laskowski > > http://hadoop.apache.org/releases.html shows 2.7.3 released on 25 January, > 2016 which should rather be 25 August, 2016. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13583) Incorporate checkcompatibility script which runs Java API Compliance Checker
Andrew Wang created HADOOP-13583: Summary: Incorporate checkcompatibility script which runs Java API Compliance Checker Key: HADOOP-13583 URL: https://issues.apache.org/jira/browse/HADOOP-13583 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.4 Reporter: Andrew Wang Assignee: Andrew Wang Based on discussion at YETUS-445, this code can't go there, but it's still very useful for release managers. A similar variant of this script has been used for a while by Apache HBase and Apache Kudu, and IMO JACC output is easier to understand than JDiff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13286) add a S3A scale test to do gunzip and linecount
[ https://issues.apache.org/jira/browse/HADOOP-13286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-13286: -- > add a S3A scale test to do gunzip and linecount > --- > > Key: HADOOP-13286 > URL: https://issues.apache.org/jira/browse/HADOOP-13286 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13286-branch-2-001.patch > > > the HADOOP-13203 patch proposal showed that there were performance problems > downstream which weren't surfacing in the current scale tests. > Trying to decompress the .gz test file and then go through it with LineReader > models a basic use case: parse a .csv.gz data source. > Add this, with metric printing -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13286) add a S3A scale test to do gunzip and linecount
[ https://issues.apache.org/jira/browse/HADOOP-13286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13286. -- Resolution: Duplicate Fix Version/s: (was: 2.8.0) > add a S3A scale test to do gunzip and linecount > --- > > Key: HADOOP-13286 > URL: https://issues.apache.org/jira/browse/HADOOP-13286 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13286-branch-2-001.patch > > > the HADOOP-13203 patch proposal showed that there were performance problems > downstream which weren't surfacing in the current scale tests. > Trying to decompress the .gz test file and then go through it with LineReader > models a basic use case: parse a .csv.gz data source. > Add this, with metric printing -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12976) s3a toString to be meaningful in logs
[
https://issues.apache.org/jira/browse/HADOOP-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-12976.
--
Resolution: Duplicate
> s3a toString to be meaningful in logs
> -
>
> Key: HADOOP-12976
> URL: https://issues.apache.org/jira/browse/HADOOP-12976
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Fix For: 2.8.0
>
>
> today's toString value is just the object ref; better to include the URL of
> the FS
> Example:
> {code}
> Cleaning filesystem org.apache.hadoop.fs.s3a.S3AFileSystem@1f069dc1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12997) s3a to pass PositionedReadable contract tests, improve readFully perf.
[ https://issues.apache.org/jira/browse/HADOOP-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-12997. -- Resolution: Duplicate Fix Version/s: (was: 2.8.0) > s3a to pass PositionedReadable contract tests, improve readFully perf. > -- > > Key: HADOOP-12997 > URL: https://issues.apache.org/jira/browse/HADOOP-12997 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > Fix s3a so that it passes the new tests in HADOOP-12994 > Also: optimise readFully so that instead of a sequence of seek-read-seek > operations, it does an opening seek and retains that position as it loops > through the data -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-12997) s3a to pass PositionedReadable contract tests, improve readFully perf.
[ https://issues.apache.org/jira/browse/HADOOP-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-12997: -- > s3a to pass PositionedReadable contract tests, improve readFully perf. > -- > > Key: HADOOP-12997 > URL: https://issues.apache.org/jira/browse/HADOOP-12997 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > Fix s3a so that it passes the new tests in HADOOP-12994 > Also: optimise readFully so that instead of a sequence of seek-read-seek > operations, it does an opening seek and retains that position as it loops > through the data -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-12976) s3a toString to be meaningful in logs
[
https://issues.apache.org/jira/browse/HADOOP-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-12976:
--
> s3a toString to be meaningful in logs
> -
>
> Key: HADOOP-12976
> URL: https://issues.apache.org/jira/browse/HADOOP-12976
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Fix For: 2.8.0
>
>
> today's toString value is just the object ref; better to include the URL of
> the FS
> Example:
> {code}
> Cleaning filesystem org.apache.hadoop.fs.s3a.S3AFileSystem@1f069dc1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12762) task: null java.lang.unsupportedoperationexception: this is supposed to be overridden by subclasses.
[ https://issues.apache.org/jira/browse/HADOOP-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-12762. -- Resolution: Cannot Reproduce > task: null java.lang.unsupportedoperationexception: this is supposed to be > overridden by subclasses. > > > Key: HADOOP-12762 > URL: https://issues.apache.org/jira/browse/HADOOP-12762 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.2 >Reporter: Padma > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-12420) While trying to access Amazon S3 through hadoop-aws(Spark basically) I was getting Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.tr
[
https://issues.apache.org/jira/browse/HADOOP-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-12420:
--
> While trying to access Amazon S3 through hadoop-aws(Spark basically) I was
> getting Exception in thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> --
>
> Key: HADOOP-12420
> URL: https://issues.apache.org/jira/browse/HADOOP-12420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Tariq Mohammad
>Assignee: Tariq Mohammad
>Priority: Minor
>
> While trying to access data stored in Amazon S3 through Apache Spark, which
> internally uses hadoop-aws jar I was getting the following exception :
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> Probable reason could be the fact that aws java sdk expects a long parameter
> for the setMultipartUploadThreshold(long multiPartThreshold) method, but
> hadoop-aws was using a parameter of type int(multiPartThreshold).
> I tried using the downloaded hadoop-aws jar and the build through its maven
> dependency, but in both the cases I encountered the same exception. Although
> I can see private long multiPartThreshold; in hadoop-aws GitHub repo, it's
> not getting reflected in the downloaded jar or in the jar created from maven
> dependency.
> Following lines in the S3AFileSystem class create this difference :
> Build from trunk :
> private long multiPartThreshold;
> this.multiPartThreshold = conf.getLong("fs.s3a.multipart.threshold",
> 2147483647L); => Line 267
> Build through maven dependency :
> private int multiPartThreshold;
> multiPartThreshold = conf.getInt(MIN_MULTIPART_THRESHOLD,
> DEFAULT_MIN_MULTIPART_THRESHOLD); => Line 249
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12420) While trying to access Amazon S3 through hadoop-aws(Spark basically) I was getting Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.tr
[
https://issues.apache.org/jira/browse/HADOOP-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-12420.
--
Resolution: Duplicate
Fix Version/s: (was: 2.8.0)
> While trying to access Amazon S3 through hadoop-aws(Spark basically) I was
> getting Exception in thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> --
>
> Key: HADOOP-12420
> URL: https://issues.apache.org/jira/browse/HADOOP-12420
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.7.1
>Reporter: Tariq Mohammad
>Assignee: Tariq Mohammad
>Priority: Minor
>
> While trying to access data stored in Amazon S3 through Apache Spark, which
> internally uses hadoop-aws jar I was getting the following exception :
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> Probable reason could be the fact that aws java sdk expects a long parameter
> for the setMultipartUploadThreshold(long multiPartThreshold) method, but
> hadoop-aws was using a parameter of type int(multiPartThreshold).
> I tried using the downloaded hadoop-aws jar and the build through its maven
> dependency, but in both the cases I encountered the same exception. Although
> I can see private long multiPartThreshold; in hadoop-aws GitHub repo, it's
> not getting reflected in the downloaded jar or in the jar created from maven
> dependency.
> Following lines in the S3AFileSystem class create this difference :
> Build from trunk :
> private long multiPartThreshold;
> this.multiPartThreshold = conf.getLong("fs.s3a.multipart.threshold",
> 2147483647L); => Line 267
> Build through maven dependency :
> private int multiPartThreshold;
> multiPartThreshold = conf.getInt(MIN_MULTIPART_THRESHOLD,
> DEFAULT_MIN_MULTIPART_THRESHOLD); => Line 249
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-12319) S3AFastOutputStream has no ability to apply backpressure
[
https://issues.apache.org/jira/browse/HADOOP-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-12319:
--
> S3AFastOutputStream has no ability to apply backpressure
>
>
> Key: HADOOP-12319
> URL: https://issues.apache.org/jira/browse/HADOOP-12319
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Colin Marc
>Priority: Critical
>
> Currently, users of S3AFastOutputStream can control memory usage with a few
> settings: {{fs.s3a.threads.core,max}}, which control the number of active
> uploads (specifically as arguments to a {{ThreadPoolExecutor}}), and
> {{fs.s3a.max.total.tasks}}, which controls the size of the feeding queue for
> the {{ThreadPoolExecutor}}.
> However, a user can get an almost *guaranteed* crash if the throughput of the
> writing job is higher than the total S3 throughput, because there is never
> any backpressure or blocking on calls to {{write}}.
> If {{fs.s3a.max.total.tasks}} is set high (the default is 1000), then
> {{write}} calls will continue to add data to the queue, which can eventually
> OOM. But if the user tries to set it lower, then writes will fail when the
> queue is full; the {{ThreadPoolExecutor}} will reject the part with
> {{java.util.concurrent.RejectedExecutionException}}.
> Ideally, calls to {{write}} should *block, not fail* when the queue is full,
> so as to apply backpressure on whatever the writing process is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12319) S3AFastOutputStream has no ability to apply backpressure
[
https://issues.apache.org/jira/browse/HADOOP-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-12319.
--
Resolution: Duplicate
Fix Version/s: (was: 2.8.0)
> S3AFastOutputStream has no ability to apply backpressure
>
>
> Key: HADOOP-12319
> URL: https://issues.apache.org/jira/browse/HADOOP-12319
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Colin Marc
>Priority: Critical
>
> Currently, users of S3AFastOutputStream can control memory usage with a few
> settings: {{fs.s3a.threads.core,max}}, which control the number of active
> uploads (specifically as arguments to a {{ThreadPoolExecutor}}), and
> {{fs.s3a.max.total.tasks}}, which controls the size of the feeding queue for
> the {{ThreadPoolExecutor}}.
> However, a user can get an almost *guaranteed* crash if the throughput of the
> writing job is higher than the total S3 throughput, because there is never
> any backpressure or blocking on calls to {{write}}.
> If {{fs.s3a.max.total.tasks}} is set high (the default is 1000), then
> {{write}} calls will continue to add data to the queue, which can eventually
> OOM. But if the user tries to set it lower, then writes will fail when the
> queue is full; the {{ThreadPoolExecutor}} will reject the part with
> {{java.util.concurrent.RejectedExecutionException}}.
> Ideally, calls to {{write}} should *block, not fail* when the queue is full,
> so as to apply backpressure on whatever the writing process is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11874) s3a can throw spurious IOEs on close()
[ https://issues.apache.org/jira/browse/HADOOP-11874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-11874. -- Resolution: Duplicate > s3a can throw spurious IOEs on close() > -- > > Key: HADOOP-11874 > URL: https://issues.apache.org/jira/browse/HADOOP-11874 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.8.0 > > > from a code review, it's clear that the issue seen in HADOOP-11851 can > surface in S3a, though with HADOOP-11570, it's less likely. It will only > happen on those cases when abort() isn't called. > The "clean" close() code path needs to catch IOEs from the wrappedStream and > call abort() in that situation too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-11874) s3a can throw spurious IOEs on close()
[ https://issues.apache.org/jira/browse/HADOOP-11874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-11874: -- > s3a can throw spurious IOEs on close() > -- > > Key: HADOOP-11874 > URL: https://issues.apache.org/jira/browse/HADOOP-11874 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.8.0 > > > from a code review, it's clear that the issue seen in HADOOP-11851 can > surface in S3a, though with HADOOP-11570, it's less likely. It will only > happen on those cases when abort() isn't called. > The "clean" close() code path needs to catch IOEs from the wrappedStream and > call abort() in that situation too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13409) Andrew's test JIRA
Andrew Wang created HADOOP-13409: Summary: Andrew's test JIRA Key: HADOOP-13409 URL: https://issues.apache.org/jira/browse/HADOOP-13409 Project: Hadoop Common Issue Type: Bug Reporter: Andrew Wang Assignee: Andrew Wang Test JIRA for JIRA interaction script -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13383) Update release notes for 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HADOOP-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13383. -- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 Thanks for the reviews Akira and Sangjin, committed to trunk and branch-3.0.0-alpha1. > Update release notes for 3.0.0-alpha1 > - > > Key: HADOOP-13383 > URL: https://issues.apache.org/jira/browse/HADOOP-13383 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13383.001.patch, HADOOP-13383.002.patch, > HADOOP-13383.003.patch, HADOOP-13383.004.patch > > > Per the release instructions (https://wiki.apache.org/hadoop/HowToRelease), > we need to update hadoop-project/src/site/markdown/index.md.vm to reflect the > right versions, new features and big improvements. > I can put together some notes for HADOOP and HDFS, depending on others for > YARN and MR. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13383) Update release notes for 3.0.0-alpha1
Andrew Wang created HADOOP-13383: Summary: Update release notes for 3.0.0-alpha1 Key: HADOOP-13383 URL: https://issues.apache.org/jira/browse/HADOOP-13383 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Blocker Per the release instructions (https://wiki.apache.org/hadoop/HowToRelease), we need to update hadoop-project/src/site/markdown/index.md.vm to reflect the right versions, new features and big improvements. I can put together some notes for HADOOP and HDFS, depending on others for YARN and MR. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13348) delete spurious 'master' branch
[ https://issues.apache.org/jira/browse/HADOOP-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13348. -- Resolution: Fixed Assignee: Andrew Wang With the help of INFRA-12223, the master branch has been deleted. Thanks Sean for reporting this issue! > delete spurious 'master' branch > --- > > Key: HADOOP-13348 > URL: https://issues.apache.org/jira/browse/HADOOP-13348 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Sean Busbey >Assignee: Andrew Wang > > Right now the git repo has a branch named 'master' in addition to our 'trunk' > branch. Since 'master' is the common-place name of the 'most recent' branch > in git repositories, this is misleading to new folks. > It looks like the branch is from ~11 months ago. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13303) Detail Informations of KMS High Avalibale
[ https://issues.apache.org/jira/browse/HADOOP-13303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13303. -- Resolution: Invalid Please use the user list for questions like this, JIRA is for tracking product defects and code changes. Thanks! > Detail Informations of KMS High Avalibale > - > > Key: HADOOP-13303 > URL: https://issues.apache.org/jira/browse/HADOOP-13303 > Project: Hadoop Common > Issue Type: Improvement > Components: ha, kms >Affects Versions: 2.7.2 >Reporter: qiushi fan > > I have some confusions of kms HA recently. > 1. we can set up multiple KMS instances behind a load balancer. Among all > these kms instances, there is only one master kms, others are slave kms. The > master kms can handle Key create/store/rollover/delete operations by directly > contacting with JCE keystore file. The slave kms can handle Key > create/store/rollover/delete operations by delegating it to the master kms. > so although we set up multiple kms, there is only one JCE keystore file, and > only the master kms can access to this file. Both the JCE keystore file and > the master kms don't have a backup. If one of them died, there is no way to > avoid losing data. > Is all of the above true? KMS doesn't have a solution to handle the failure > of master kms and JCE keystore file? > 2. I heard another way to achieve kms HA: make use of > LoadBalancingKMSClientProvider. But I can't find detail informations of > LoadBalancingKMSClientProvider. So why the LoadBalancingKMSClientProvider > can achieve kms HA? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13175. -- Resolution: Fixed Hadoop Flags: Incompatible change Fix Version/s: 3.0.0-alpha1 Release Note: The hadoop-ant module in hadoop-tools has been removed. Committed to trunk, thanks Chris for the patch, Jason and Akira for reviewing! > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Chris Douglas > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HADOOP-13175: -- Assignee: Chris Douglas Reopening based on Jason's latest comment. FWIW patch LGTM +1. Chris, anything else we need to do before checking this in for 3.0? > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas >Assignee: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11858) [JDK8] Set minimum version of Hadoop 3 to JDK 8
[ https://issues.apache.org/jira/browse/HADOOP-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-11858. -- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 Target Version/s: (was: ) Committed to trunk. Thanks Robert for the patch and everyone for reviewing! > [JDK8] Set minimum version of Hadoop 3 to JDK 8 > --- > > Key: HADOOP-11858 > URL: https://issues.apache.org/jira/browse/HADOOP-11858 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-11858.001.patch, HADOOP-11858.002.patch, > HADOOP-11858.003.patch > > > Set minimum version of trunk to JDK 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12868) Fix hadoop-openstack undeclared and unused dependencies
[ https://issues.apache.org/jira/browse/HADOOP-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-12868. -- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: (was: ) Committed to trunk, branch-2, branch-2.8. Thanks for the patch [~iwasakims]! > Fix hadoop-openstack undeclared and unused dependencies > --- > > Key: HADOOP-12868 > URL: https://issues.apache.org/jira/browse/HADOOP-12868 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Masatake Iwasaki > Fix For: 2.8.0 > > Attachments: HADOOP-12868.001.patch > > > Attempting to compile openstack on a fairly fresh maven repo fails due to > commons-httpclient not being a declared dependency. After that is fixed, > doing a maven dependency:analyze shows other problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13142) Change project version from 3.0.0 to 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HADOOP-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13142. -- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 Thanks for the review Colin, committed to trunk. > Change project version from 3.0.0 to 3.0.0-alpha1 > - > > Key: HADOOP-13142 > URL: https://issues.apache.org/jira/browse/HADOOP-13142 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: hadoop-13142.001.patch > > > We want to rename 3.0.0 to 3.0.0-alpha1 for the first alpha release. However, > the version number is also encoded outside of the pom.xml's, so we need to > update these too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13142) Change project version from 3.0.0 to 3.0.0-alpha1
Andrew Wang created HADOOP-13142: Summary: Change project version from 3.0.0 to 3.0.0-alpha1 Key: HADOOP-13142 URL: https://issues.apache.org/jira/browse/HADOOP-13142 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Blocker We want to rename 3.0.0 to 3.0.0-alpha1 for the first alpha release. However, the version number is also encoded outside of the pom.xml's, so we need to update these too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13141) Mini HDFS Cluster fails to start on trunk
[ https://issues.apache.org/jira/browse/HADOOP-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-13141. -- Resolution: Fixed > Mini HDFS Cluster fails to start on trunk > - > > Key: HADOOP-13141 > URL: https://issues.apache.org/jira/browse/HADOOP-13141 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaobing Zhou > > It's been noticed that Mini HDFS Cluster fails to start on trunk, blocking > unit tests and Jenkins. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11793) Update create-release for releasedocmaker.py
[ https://issues.apache.org/jira/browse/HADOOP-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-11793. -- Resolution: Duplicate Duping this one, since it's handled by the create-release rewrite in HADOOP-12892. Thanks all! > Update create-release for releasedocmaker.py > > > Key: HADOOP-11793 > URL: https://issues.apache.org/jira/browse/HADOOP-11793 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 2.8.0 >Reporter: Allen Wittenauer >Assignee: ramtin > Attachments: HADOOP-11793.001.patch > > > With the commit of HADOOP-11731, the changelog and release note data is now > automated with the build. The create-release script needs to do the correct > thing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13043) Add LICENSE.txt entries for bundled javascript dependencies
Andrew Wang created HADOOP-13043: Summary: Add LICENSE.txt entries for bundled javascript dependencies Key: HADOOP-13043 URL: https://issues.apache.org/jira/browse/HADOOP-13043 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.4 Reporter: Andrew Wang Assignee: Andrew Wang None of our bundled javascript dependencies are mentioned in LICENSE.txt. Let's fix that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-13042) Restore lost leveldbjni LICENSE and NOTICE changes
Andrew Wang created HADOOP-13042: Summary: Restore lost leveldbjni LICENSE and NOTICE changes Key: HADOOP-13042 URL: https://issues.apache.org/jira/browse/HADOOP-13042 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.4 Reporter: Andrew Wang Assignee: Andrew Wang As noted on HADOOP-12893, we lost the leveldbjni related NOTICE and LICENSE updates done in YARN-1704 when HADOOP-10956 was committed. Let's restore them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11792) Remove all of the CHANGES.txt files
[ https://issues.apache.org/jira/browse/HADOOP-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-11792. -- Resolution: Fixed Fix Version/s: 2.8.0 Since we just resolved HADOOP-12651, I blew away CHANGES.txt on trunk, branch-2, and branch-2.8. Will send a notice to common-dev, happy committing all! > Remove all of the CHANGES.txt files > --- > > Key: HADOOP-11792 > URL: https://issues.apache.org/jira/browse/HADOOP-11792 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Andrew Wang > Fix For: 2.8.0 > > > With the commit of HADOOP-11731, the CHANGES.txt files are now EOLed. We > should remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12713) Disable spurious checkstyle checks
Andrew Wang created HADOOP-12713: Summary: Disable spurious checkstyle checks Key: HADOOP-12713 URL: https://issues.apache.org/jira/browse/HADOOP-12713 Project: Hadoop Common Issue Type: Improvement Reporter: Andrew Wang Assignee: Andrew Wang Some of the checkstyle checks are not realistic (like the line length), leading to spurious -1 in precommit. Let's disable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12367) Move TestFileUtil's test resources to resources folder
Andrew Wang created HADOOP-12367: Summary: Move TestFileUtil's test resources to resources folder Key: HADOOP-12367 URL: https://issues.apache.org/jira/browse/HADOOP-12367 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.7.1 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Little cleanup. Right now we do an antrun step to copy the tar and tgz from the source folder to target folder. We can skip this by just putting it in the resources folder like all the other test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
