[jira] [Resolved] (HADOOP-15208) DistCp to offer -xtrack option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved HADOOP-15208. -- Resolution: Duplicate Fix Version/s: (was: 3.1.0) > DistCp to offer -xtrack option to save src/dest filesets as > alternative to delete() > -- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch, HADOOP-15208-003.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15208) DistCp to offer -xtrack option to save src/dest filesets as alternative to delete()
[ https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened HADOOP-15208: -- Reopening and closing this instead as a dup of HADOOP-15209 as I can't find any patch for this in 3.1.0 for this JIRA. Revert back if this is incorrect. > DistCp to offer -xtrack option to save src/dest filesets as > alternative to delete() > -- > > Key: HADOOP-15208 > URL: https://issues.apache.org/jira/browse/HADOOP-15208 > Project: Hadoop Common > Issue Type: New Feature > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, > HADOOP-15208-002.patch, HADOOP-15208-003.patch > > > There are opportunities to improve distcp delete performance and scalability > with object stores, but you need to test with production datasets to > determine if the optimizations work, don't run out of memory, etc. > By adding the option to save the sequence files of source, dest listings, > people (myself included) can experiment with different strategies before > trying to commit one which doesn't scale -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14974) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation fails in trunk
[
https://issues.apache.org/jira/browse/HADOOP-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli resolved HADOOP-14974.
--
Resolution: Duplicate
Fix Version/s: (was: 3.1.0)
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
> fails in trunk
> ---
>
> Key: HADOOP-14974
> URL: https://issues.apache.org/jira/browse/HADOOP-14974
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: John Zhuge
>Priority: Blocker
>
> {code}
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> QueueMetrics,q0=root already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:239)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:141)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:90)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:267)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:639)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:331)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:391)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:756)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1313)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:161)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testExcessReservationThanNodeManagerCapacity(TestContainerAllocation.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14974) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation fails in trunk
[
https://issues.apache.org/jira/browse/HADOOP-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reopened HADOOP-14974:
--
HADOOP-14954 never made it to a release. And there's no other patch for this
JIRA in the 3.1.0 release. Reopening and closing this instead as a dup.
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
> fails in trunk
> ---
>
> Key: HADOOP-14974
> URL: https://issues.apache.org/jira/browse/HADOOP-14974
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: John Zhuge
>Priority: Blocker
> Fix For: 3.1.0
>
>
> {code}
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> QueueMetrics,q0=root already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:239)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:141)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:90)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:267)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:639)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:331)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:391)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:756)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1313)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:161)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testExcessReservationThanNodeManagerCapacity(TestContainerAllocation.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14714) handle InternalError in bulk object delete through retries
[ https://issues.apache.org/jira/browse/HADOOP-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved HADOOP-14714. -- Resolution: Duplicate Fix Version/s: (was: 3.1.0) > handle InternalError in bulk object delete through retries > -- > > Key: HADOOP-14714 > URL: https://issues.apache.org/jira/browse/HADOOP-14714 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > There's some more detail appearing on HADOOP-11572 about the errors seen > here; sounds like its large fileset related (or just probability working > against you). Most importantly: retries may make it go away. > Proposed: implement a retry policy. > Issue: delete is not idempotent, not if someone else adds things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14714) handle InternalError in bulk object delete through retries
[ https://issues.apache.org/jira/browse/HADOOP-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened HADOOP-14714: -- Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any patch for this in the source-code. Please revert back if this is incorrect. > handle InternalError in bulk object delete through retries > -- > > Key: HADOOP-14714 > URL: https://issues.apache.org/jira/browse/HADOOP-14714 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > There's some more detail appearing on HADOOP-11572 about the errors seen > here; sounds like its large fileset related (or just probability working > against you). Most importantly: retries may make it go away. > Proposed: implement a retry policy. > Issue: delete is not idempotent, not if someone else adds things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14585) Ensure controls in-place to prevent clients with significant clock skews pruning aggressively
[
https://issues.apache.org/jira/browse/HADOOP-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14585:
-
Fix Version/s: (was: 3.1.0)
Removing fix-version from this open JIRA - it is instead set during commit-time.
> Ensure controls in-place to prevent clients with significant clock skews
> pruning aggressively
> -
>
> Key: HADOOP-14585
> URL: https://issues.apache.org/jira/browse/HADOOP-14585
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
>Priority: Minor
>
> From discussion on HADOOP-14499:
> {quote}
> bear in mind that we can't guarantee that the clocks of all clients are in
> sync; you don't want a client whose TZ setting is wrong to aggressively prune
> things. Had that happen in production with files in shared filestore. This is
> why ant -diagnostics checks time consistency with temp files...
> {quote}
> {quote}
> temp files work on a shared FS. AWS is actually somewhat sensitive to clocks:
> if your VM is too far out of time then auth actually fails, its ~+-15
> minutes. There's some stuff in the Java SDK to actually calculate and adjust
> clock skew, presumably parsing the timestamp of a failure, calculating the
> difference and retrying. Which means that the field in SDKGlobalConfiguration
> could help identify the difference between local time and AWS time.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14585) Ensure controls in-place to prevent clients with significant clock skews pruning aggressively
[
https://issues.apache.org/jira/browse/HADOOP-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14585:
-
Removing fix-version from this open JIRA - it is instead set during commit-time.
> Ensure controls in-place to prevent clients with significant clock skews
> pruning aggressively
> -
>
> Key: HADOOP-14585
> URL: https://issues.apache.org/jira/browse/HADOOP-14585
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
>Priority: Minor
>
> From discussion on HADOOP-14499:
> {quote}
> bear in mind that we can't guarantee that the clocks of all clients are in
> sync; you don't want a client whose TZ setting is wrong to aggressively prune
> things. Had that happen in production with files in shared filestore. This is
> why ant -diagnostics checks time consistency with temp files...
> {quote}
> {quote}
> temp files work on a shared FS. AWS is actually somewhat sensitive to clocks:
> if your VM is too far out of time then auth actually fails, its ~+-15
> minutes. There's some stuff in the Java SDK to actually calculate and adjust
> clock skew, presumably parsing the timestamp of a failure, calculating the
> difference and retrying. Which means that the field in SDKGlobalConfiguration
> could help identify the difference between local time and AWS time.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14531) [Umbrella] Improve S3A error handling & reporting
[ https://issues.apache.org/jira/browse/HADOOP-14531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-14531: - Summary: [Umbrella] Improve S3A error handling & reporting (was: Improve S3A error handling & reporting) > [Umbrella] Improve S3A error handling & reporting > - > > Key: HADOOP-14531 > URL: https://issues.apache.org/jira/browse/HADOOP-14531 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.1.0 > > > Improve S3a error handling and reporting > this includes > # looking at error codes and translating to more specific exceptions > # better retry logic where present > # adding retry logic where not present > # more diagnostics in exceptions > # docs > Overall goals > * things that can be retried and will go away are retried for a bit > * things that don't go away when retried failfast (302, no auth, unknown > host, connection refused) > * meaningful exceptions are built in translate exception > * diagnostics are included, where possible > * our troubleshooting docs are expanded with new failures we encounter > AWS S3 error codes: > http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14381) S3AUtils.translateException to map 503 reponse to => throttling failure
[
https://issues.apache.org/jira/browse/HADOOP-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reopened HADOOP-14381:
--
Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any
patch for this in the source-code. Please revert back if this is incorrect.
> S3AUtils.translateException to map 503 reponse to => throttling failure
> ---
>
> Key: HADOOP-14381
> URL: https://issues.apache.org/jira/browse/HADOOP-14381
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> When AWS S3 returns "503", it means that the overall set of requests on a
> part of an S3 bucket exceeds the permitted limit; the client(s) need to
> throttle back or away for some rebalancing to complete.
> The aws SDK retries 3 times on a 503, but then throws it up. Our code doesn't
> do anything with that other than create a generic {{AWSS3IOException}}.
> Proposed
> * add a new exception, {{AWSOverloadedException}}
> * raise it on a 503 from S3 (& for s3guard, on DDB complaints)
> * have it include a link to a wiki page on the topic, as well as the path
> * and any other diags
> Code talking to S3 may then be able to catch this and choose to react. Some
> retry with exponential backoff is the obvious option. Failing, well, that
> could trigger task reattempts at that part of the query, then job retry
> —which will again fail, *unless the number of tasks run in parallel is
> reduced*
> As this throttling is across all clients talking to the same part of a
> bucket, fixing it is potentially a high level option. We can at least start
> by reporting things better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14381) S3AUtils.translateException to map 503 reponse to => throttling failure
[
https://issues.apache.org/jira/browse/HADOOP-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli resolved HADOOP-14381.
--
Resolution: Duplicate
Fix Version/s: (was: 3.1.0)
> S3AUtils.translateException to map 503 reponse to => throttling failure
> ---
>
> Key: HADOOP-14381
> URL: https://issues.apache.org/jira/browse/HADOOP-14381
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> When AWS S3 returns "503", it means that the overall set of requests on a
> part of an S3 bucket exceeds the permitted limit; the client(s) need to
> throttle back or away for some rebalancing to complete.
> The aws SDK retries 3 times on a 503, but then throws it up. Our code doesn't
> do anything with that other than create a generic {{AWSS3IOException}}.
> Proposed
> * add a new exception, {{AWSOverloadedException}}
> * raise it on a 503 from S3 (& for s3guard, on DDB complaints)
> * have it include a link to a wiki page on the topic, as well as the path
> * and any other diags
> Code talking to S3 may then be able to catch this and choose to react. Some
> retry with exponential backoff is the obvious option. Failing, well, that
> could trigger task reattempts at that part of the query, then job retry
> —which will again fail, *unless the number of tasks run in parallel is
> reduced*
> As this throttling is across all clients talking to the same part of a
> bucket, fixing it is potentially a high level option. We can at least start
> by reporting things better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14325) [Umbrella] Stabilise S3A Server Side Encryption
[ https://issues.apache.org/jira/browse/HADOOP-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-14325: - Summary: [Umbrella] Stabilise S3A Server Side Encryption (was: Stabilise S3A Server Side Encryption) > [Umbrella] Stabilise S3A Server Side Encryption > --- > > Key: HADOOP-14325 > URL: https://issues.apache.org/jira/browse/HADOOP-14325 > Project: Hadoop Common > Issue Type: Task > Components: documentation, fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Major > Fix For: 3.1.0 > > > Round off the S3 SSE encryption support with everything needed to safely ship > it. > The core code is in, along with tests, so this covers the details > * docs with examples, including JCEKS files > * keeping secrets secret > * any more tests, including scale ones (huge file, rename) > * I'll add a KMS test to my (github) spark suite -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409052#comment-16409052
]
ASF GitHub Bot commented on HADOOP-15230:
-
Github user bak12 commented on the issue:
https://github.com/apache/hadoop/pull/340
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.007.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13205) S3A to support custom retry policies; failfast on unknown host
[ https://issues.apache.org/jira/browse/HADOOP-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved HADOOP-13205. -- Resolution: Duplicate Fix Version/s: (was: 3.1.0) > S3A to support custom retry policies; failfast on unknown host > -- > > Key: HADOOP-13205 > URL: https://issues.apache.org/jira/browse/HADOOP-13205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Noticed today that when connections are down, S3A retries on > UnknownHostExceptions logging noisily in the process. > # it should be possible to define or customize retry policies for an FS > instance (fail fast, exponential backoff, etc) > # we may want to explicitly have a fail-fast-if-offline retry policy, > catching the common connectivity ones. > Testing will be fun here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13205) S3A to support custom retry policies; failfast on unknown host
[ https://issues.apache.org/jira/browse/HADOOP-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened HADOOP-13205: -- Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any patch for this JIRA in the source-code. Please revert back if this is incorrect. > S3A to support custom retry policies; failfast on unknown host > -- > > Key: HADOOP-13205 > URL: https://issues.apache.org/jira/browse/HADOOP-13205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0 > > > Noticed today that when connections are down, S3A retries on > UnknownHostExceptions logging noisily in the process. > # it should be possible to define or customize retry policies for an FS > instance (fail fast, exponential backoff, etc) > # we may want to explicitly have a fail-fast-if-offline retry policy, > catching the common connectivity ones. > Testing will be fun here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli resolved HADOOP-13811.
--
Resolution: Duplicate
Fix Version/s: (was: 3.1.0)
> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document
> destined for handler class}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reopened HADOOP-13811:
--
Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any
patch for this JIRA in the source-code. Please revert back if this is incorrect.
> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document
> destined for handler class}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14303) Review retry logic on all S3 SDK calls, implement where needed
[ https://issues.apache.org/jira/browse/HADOOP-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reopened HADOOP-14303: -- Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any patch for this in the source-code. Please revert back if this is incorrect. > Review retry logic on all S3 SDK calls, implement where needed > -- > > Key: HADOOP-14303 > URL: https://issues.apache.org/jira/browse/HADOOP-14303 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > AWS S3, IAM, KMS, DDB etc all throttle callers: the S3A code needs to handle > this without failing, as if it slows down its requests it can recover. > 1. Look at all the places where we are calling S3A via the AWS SDK and make > sure we are retrying with some backoff & jitter policy, ideally something > unified. This must be more systematic than the case-by-case, > problem-by-problem strategy we are implicitly using. > 2. Many of the AWS S3 SDK calls do implement retry (e.g PUT/multipart PUT), > but we need to check the other parts of the process: login, initiate/complete > MPU, ... > Related > HADOOP-13811 Failed to sanitize XML document destined for handler class > HADOOP-13664 S3AInputStream to use a retry policy on read failures > This stuff is all hard to test. A key need is to be able to differentiate > recoverable throttle & network failures from unrecoverable problems like: > auth, network config (e.g bad endpoint), etc. > May be the opportunity to add a faulting subclass of Amazon S3 client which > can be configured in IT Tests to fail at specific points. Ryan Blue's mcok S3 > client does this in HADOOP-13786, but it is for 100% mock. I'm thinking of > something with similar fault raising, but in front of the real S3A client -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14303) Review retry logic on all S3 SDK calls, implement where needed
[ https://issues.apache.org/jira/browse/HADOOP-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved HADOOP-14303. -- Resolution: Duplicate Fix Version/s: (was: 3.1.0) > Review retry logic on all S3 SDK calls, implement where needed > -- > > Key: HADOOP-14303 > URL: https://issues.apache.org/jira/browse/HADOOP-14303 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > AWS S3, IAM, KMS, DDB etc all throttle callers: the S3A code needs to handle > this without failing, as if it slows down its requests it can recover. > 1. Look at all the places where we are calling S3A via the AWS SDK and make > sure we are retrying with some backoff & jitter policy, ideally something > unified. This must be more systematic than the case-by-case, > problem-by-problem strategy we are implicitly using. > 2. Many of the AWS S3 SDK calls do implement retry (e.g PUT/multipart PUT), > but we need to check the other parts of the process: login, initiate/complete > MPU, ... > Related > HADOOP-13811 Failed to sanitize XML document destined for handler class > HADOOP-13664 S3AInputStream to use a retry policy on read failures > This stuff is all hard to test. A key need is to be able to differentiate > recoverable throttle & network failures from unrecoverable problems like: > auth, network config (e.g bad endpoint), etc. > May be the opportunity to add a faulting subclass of Amazon S3 client which > can be configured in IT Tests to fail at specific points. Ryan Blue's mcok S3 > client does this in HADOOP-13786, but it is for 100% mock. I'm thinking of > something with similar fault raising, but in front of the real S3A client -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409031#comment-16409031
]
genericqa commented on HADOOP-15331:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 9s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 55s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915571/HADOOP-15331.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9177e2a753c0 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d898ab |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14370/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14370/testReport/ |
| Max. process+thread count | 1371 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14370/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[
https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409010#comment-16409010
]
genericqa commented on HADOOP-14067:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 31s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
58s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-14067 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915584/HADOOP-14067.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 029bc91683e9 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d898ab |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14368/testReport/ |
| Max. process+thread count | 1413 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14368/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> VersionInfo should load
[jira] [Updated] (HADOOP-15335) Support xxxxxxx:xxx/stacks print lock info and more useful attribute of thread info
[ https://issues.apache.org/jira/browse/HADOOP-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HADOOP-15335: -- Status: Patch Available (was: Open) > Support xxx:xxx/stacks print lock info and more useful attribute of > thread info > --- > > Key: HADOOP-15335 > URL: https://issues.apache.org/jira/browse/HADOOP-15335 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: Yiran Wu >Priority: Major > Attachments: HADOOP-15335.001.patch > > > Print stack information and other info show in WebUI > http://namenode:50070/stacks?contentionTracing=true > ``` > Thread 2 (Reference Handler): > State: WAITING > Blocked count: 8 > Waited count: 5 > Thread CpuTime: 591 > Thread UserTime: 5754000 > Thread allocatedBytes: 0 > Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > Thread 1 (main): > State: WAITING > Blocked count: 4 > Waited count: 2 > Thread CpuTime: 2563937000 > Thread UserTime: 977000 > Thread allocatedBytes: 229115520 > Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > org.apache.hadoop.ipc.Server.join(Server.java:2498) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) > org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) > - > Locks info: > - > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy > Thread 43 (IPC Server handler 7 on 8020) > Waiting thread num: 6 > Stack: > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:422) > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) > java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW > java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW > org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW > ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15335) Support xxxxxxx:xxx/stacks print lock info and more useful attribute of thread info
[ https://issues.apache.org/jira/browse/HADOOP-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HADOOP-15335: -- Attachment: HADOOP-15335.001.patch > Support xxx:xxx/stacks print lock info and more useful attribute of > thread info > --- > > Key: HADOOP-15335 > URL: https://issues.apache.org/jira/browse/HADOOP-15335 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: Yiran Wu >Priority: Major > Attachments: HADOOP-15335.001.patch > > > Print stack information and other info show in WebUI > http://namenode:50070/stacks?contentionTracing=true > ``` > Thread 2 (Reference Handler): > State: WAITING > Blocked count: 8 > Waited count: 5 > Thread CpuTime: 591 > Thread UserTime: 5754000 > Thread allocatedBytes: 0 > Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > Thread 1 (main): > State: WAITING > Blocked count: 4 > Waited count: 2 > Thread CpuTime: 2563937000 > Thread UserTime: 977000 > Thread allocatedBytes: 229115520 > Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > org.apache.hadoop.ipc.Server.join(Server.java:2498) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) > org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) > - > Locks info: > - > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy > Thread 43 (IPC Server handler 7 on 8020) > Waiting thread num: 6 > Stack: > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:422) > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) > java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW > java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW > org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW > ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15335) Support xxxxxxx:xxx/Stacks print lock info and more useful attribute of thread info
[ https://issues.apache.org/jira/browse/HADOOP-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HADOOP-15335: -- Description: Print stack information and other info show in WebUI http://namenode:50070/stacks?contentionTracing=true ``` Thread 2 (Reference Handler): State: WAITING Blocked count: 8 Waited count: 5 Thread CpuTime: 591 Thread UserTime: 5754000 Thread allocatedBytes: 0 Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: WAITING Blocked count: 4 Waited count: 2 Thread CpuTime: 2563937000 Thread UserTime: 977000 Thread allocatedBytes: 229115520 Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.ipc.Server.join(Server.java:2498) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) - Locks info: - java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy Thread 43 (IPC Server handler 7 on 8020) Waiting thread num: 6 Stack: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW ``` was: Print stack information and other info show in WebUI ``` Thread 2 (Reference Handler): State: WAITING Blocked count: 8 Waited count: 5 Thread CpuTime: 591 Thread UserTime: 5754000 Thread allocatedBytes: 0 Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: WAITING Blocked count: 4 Waited count: 2 Thread CpuTime: 2563937000 Thread UserTime: 977000 Thread allocatedBytes: 229115520 Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.ipc.Server.join(Server.java:2498) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) - Locks info: - java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy Thread 43 (IPC Server handler 7 on 8020) Waiting thread num: 6 Stack: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
[jira] [Updated] (HADOOP-15335) Support xxxxxxx:xxx/stacks print lock info and more useful attribute of thread info
[ https://issues.apache.org/jira/browse/HADOOP-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HADOOP-15335: -- Summary: Support xxx:xxx/stacks print lock info and more useful attribute of thread info (was: Support xxx:xxx/Stacks print lock info and more useful attribute of thread info) > Support xxx:xxx/stacks print lock info and more useful attribute of > thread info > --- > > Key: HADOOP-15335 > URL: https://issues.apache.org/jira/browse/HADOOP-15335 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: Yiran Wu >Priority: Major > > Print stack information and other info show in WebUI > http://namenode:50070/stacks?contentionTracing=true > ``` > Thread 2 (Reference Handler): > State: WAITING > Blocked count: 8 > Waited count: 5 > Thread CpuTime: 591 > Thread UserTime: 5754000 > Thread allocatedBytes: 0 > Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > Thread 1 (main): > State: WAITING > Blocked count: 4 > Waited count: 2 > Thread CpuTime: 2563937000 > Thread UserTime: 977000 > Thread allocatedBytes: 229115520 > Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 > Blocked by -1 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:502) > org.apache.hadoop.ipc.Server.join(Server.java:2498) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) > org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) > - > Locks info: > - > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy > Thread 43 (IPC Server handler 7 on 8020) > Waiting thread num: 6 > Stack: > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:422) > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) > java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW > java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW > org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW > ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15335) Support xxxxxxx:xxx/Stacks print lock info and more useful attribute of thread info
Yiran Wu created HADOOP-15335: - Summary: Support xxx:xxx/Stacks print lock info and more useful attribute of thread info Key: HADOOP-15335 URL: https://issues.apache.org/jira/browse/HADOOP-15335 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.0.0 Reporter: Yiran Wu Print stack information and other info show in WebUI ``` Thread 2 (Reference Handler): State: WAITING Blocked count: 8 Waited count: 5 Thread CpuTime: 591 Thread UserTime: 5754000 Thread allocatedBytes: 0 Waiting on java.lang.ref.Reference$Lock@4b3ed2f0 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: WAITING Blocked count: 4 Waited count: 2 Thread CpuTime: 2563937000 Thread UserTime: 977000 Thread allocatedBytes: 229115520 Waiting on org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 Blocked by -1 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.ipc.Server.join(Server.java:2498) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.join(NameNodeRpcServer.java:442) org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:865) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1573) - Locks info: - java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@cd6c71a lockedBy Thread 43 (IPC Server handler 7 on 8020) Waiting thread num: 6 Stack: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3665) org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:868) org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:583) org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1803) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2072) java.lang.ref.ReferenceQueue$Lock@31bcf236 lockedBy UNKNOW java.lang.ref.Reference$Lock@4b3ed2f0 lockedBy UNKNOW org.apache.hadoop.ipc.ProtobufRpcEngine$Server@442a2e48 lockedBy UNKNOW ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15334) Upgrade Maven surefire plugin
[
https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408981#comment-16408981
]
genericqa commented on HADOOP-15334:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 43s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
10s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15334 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915561/HADOOP-15334.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 406c33ab16be 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d898ab |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14369/testReport/ |
| Max. process+thread count | 440 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14369/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Upgrade Maven surefire plugin
> -
>
> Key: HADOOP-15334
> URL: https://issues.apache.org/jira/browse/HADOOP-15334
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Major
> Attachments: HADOOP-15334.01.patch
>
>
> Recent versions of the surefire plugin
[jira] [Commented] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[ https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408843#comment-16408843 ] Thejas M Nair commented on HADOOP-14067: Uploading 03.patch for real this time! > VersionInfo should load version-info.properties from its own classloader > > > Key: HADOOP-14067 > URL: https://issues.apache.org/jira/browse/HADOOP-14067 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Attachments: HADOOP-14067.01.patch, HADOOP-14067.01.patch, > HADOOP-14067.02.patch, HADOOP-14067.03.patch > > > org.apache.hadoop.util.VersionInfo loads the version-info.properties file via > the current thread classloader. > However, in case of applications that are using hadoop classes dynamically > (eg jdbc based tools such as SQuirreL SQL) the current thread might not be > the one that loaded the hadoop classes including VersionInfo, and it would > fail to fine the properties file. > The right place to look for the properties file is in the classloader of > VersionInfo class, as right version is the one that is associated with rest > of the loaded hadoop classes, and not necessarily the one in current thread > classloader. > Created a related jira - HADOOP-14066 to make methods to get version via > VersionInfo a public api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[ https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HADOOP-14067: --- Attachment: HADOOP-14067.03.patch > VersionInfo should load version-info.properties from its own classloader > > > Key: HADOOP-14067 > URL: https://issues.apache.org/jira/browse/HADOOP-14067 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Attachments: HADOOP-14067.01.patch, HADOOP-14067.01.patch, > HADOOP-14067.02.patch, HADOOP-14067.03.patch > > > org.apache.hadoop.util.VersionInfo loads the version-info.properties file via > the current thread classloader. > However, in case of applications that are using hadoop classes dynamically > (eg jdbc based tools such as SQuirreL SQL) the current thread might not be > the one that loaded the hadoop classes including VersionInfo, and it would > fail to fine the properties file. > The right place to look for the properties file is in the classloader of > VersionInfo class, as right version is the one that is associated with rest > of the loaded hadoop classes, and not necessarily the one in current thread > classloader. > Created a related jira - HADOOP-14066 to make methods to get version via > VersionInfo a public api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408842#comment-16408842 ] Arpit Agarwal commented on HADOOP-15334: bq. Can we bring this back to branch-2, also? Sure. Just need a binding +1 (thanks for the review though, [~bharatviswa]). > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15334: --- Target Version/s: 3.1.0, 2.10.0 (was: 3.1.0) > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15334) Upgrade Maven surefire plugin
[
https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408837#comment-16408837
]
genericqa commented on HADOOP-15334:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
34m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15334 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915561/HADOOP-15334.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 6c5a0ac9fc16 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5aa7052 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14367/testReport/ |
| Max. process+thread count | 365 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14367/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Upgrade Maven surefire plugin
> -
>
> Key: HADOOP-15334
> URL: https://issues.apache.org/jira/browse/HADOOP-15334
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Major
> Attachments: HADOOP-15334.01.patch
>
>
> Recent versions of the surefire
[jira] [Commented] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408834#comment-16408834 ] Chris Douglas commented on HADOOP-15334: Can we bring this back to branch-2, also? > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[
https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408822#comment-16408822
]
genericqa commented on HADOOP-14067:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 12 unchanged - 0 fixed = 14 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 4s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 38s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestDiskChecker |
| | hadoop.util.TestReadWriteDiskValidator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-14067 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915231/HADOOP-14067.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 28aa32496d31 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5aa7052 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14366/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Updated] (HADOOP-13271) Intermittent failure of TestS3AContractRootDir.testListEmptyRootDirectory
[
https://issues.apache.org/jira/browse/HADOOP-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-13271:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> Intermittent failure of TestS3AContractRootDir.testListEmptyRootDirectory
> -
>
> Key: HADOOP-13271
> URL: https://issues.apache.org/jira/browse/HADOOP-13271
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> I'm seeing an intermittent failure of
> {{TestS3AContractRootDir.testListEmptyRootDirectory}}
> The sequence of : deleteFiles(listStatus(Path("/)")) is failing because the
> file to delete is root ...yet the code is passing in the children of /, not /
> itself.
> hypothesis: when you call listStatus on an empty root dir, you get a file
> entry back that says isFile, not isDirectory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15095) S3a committer factory to warn when default FileOutputFormat committer is created
[ https://issues.apache.org/jira/browse/HADOOP-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-15095: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > S3a committer factory to warn when default FileOutputFormat committer is > created > > > Key: HADOOP-15095 > URL: https://issues.apache.org/jira/browse/HADOOP-15095 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Minor > > The S3ACommitterFactory should warn when the classic FileOutputCommitter is > used (i.e. the client is not configured to use a new one). Something like > "this committer is neither fast nor guaranteed to be correct. See $URL" where > URL is a pointer to something (wiki? hadoop docs?). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13968) S3a FS to support "__magic" path for the special "unmaterialized" writes
[
https://issues.apache.org/jira/browse/HADOOP-13968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-13968:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> S3a FS to support "__magic" path for the special "unmaterialized" writes
>
>
> Key: HADOOP-13968
> URL: https://issues.apache.org/jira/browse/HADOOP-13968
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> S3AFileSystem to add support for a special path, such as
> {{.temp_pending_put/}} or similar, which, when used as the base of a path,
> indicates that the file is actually to be saved to the parent dir, but only
> via a delayed put commit operation.
> At the same time, we may need blocks on some normal fileIO ops under these
> dirs, especially rename and delete, as this would cause serious problems
> including data loss and large bills for pending data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15241) transient failure of ITestCommitOperations.testCommitEmptyFile; fault injection related
[
https://issues.apache.org/jira/browse/HADOOP-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-15241:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> transient failure of ITestCommitOperations.testCommitEmptyFile; fault
> injection related
> ---
>
> Key: HADOOP-15241
> URL: https://issues.apache.org/jira/browse/HADOOP-15241
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Reporter: Steve Loughran
>Priority: Minor
>
> Parallel 12 thread run on a machine overloaded with other work; transient
> failure of a test expecting fault injection to raise 2 exceptions before
> success; a 3rd one was caught.
> {code}
> [ERROR]
> testCommitEmptyFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations)
> Time elapsed: 10.805 s <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Completing multipart
> commit on
> fork-00010/test/DELAY_LISTING_ME/testCommitEmptyFile/empty-commit.txt:
> com.amazonaws.AmazonServiceException: throttled count = 3 (Service: null;
> Status Code: 503; Error Code: null; Request ID: null):null: throttled count =
> 3 (Service: null; Status Code: 503; Error Code: null; Request ID: null)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14971) Merge S3A committers into trunk
[ https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-14971: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > Merge S3A committers into trunk > --- > > Key: HADOOP-14971 > URL: https://issues.apache.org/jira/browse/HADOOP-14971 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-13786-040.patch, HADOOP-13786-041.patch > > > Merge the HADOOP-13786 committer into trunk. This branch is being set up as a > github PR for review there & to keep it out the mailboxes of the watchers on > the main JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-15269:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14606) S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward seek
[
https://issues.apache.org/jira/browse/HADOOP-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14606:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward
> seek
> ---
>
> Key: HADOOP-14606
> URL: https://issues.apache.org/jira/browse/HADOOP-14606
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Critical
>
> There's some hints in the InputStream docs that {{skip(n)}} may skip bytes. Codepaths only seem to do this if read() returns -1, meaning end of
> stream is reached.
> If that happens when doing a forward seek via skip, then we have got our
> numbers wrong and are in trouble. Look for a negative response, log @ ERROR
> and revert to a close/reopen seek to an absolute position.
> *I have no evidence of this acutally occurring*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14161) Failed to rename file in S3A during FileOutputFormat commitTask
[
https://issues.apache.org/jira/browse/HADOOP-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14161:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> Failed to rename file in S3A during FileOutputFormat commitTask
> ---
>
> Key: HADOOP-14161
> URL: https://issues.apache.org/jira/browse/HADOOP-14161
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.0, 2.7.1, 2.7.2, 2.7.3
> Environment: spark 2.0.2 with mesos
> hadoop 2.7.2
>Reporter: Luke Miner
>Priority: Minor
>
> I'm getting non deterministic rename errors while writing to S3 using spark
> and hadoop. The proper permissions are set and this only happens
> occasionally. It can happen on a job that is as simple as reading in json,
> repartitioning and then writing out. After this failure occurs, the overall
> job hangs indefinitely.
> {code}
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:261)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
> at org.apache.spark.scheduler.Task.run(Task.scala:86)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed to commit task
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer.org$apache$spark$sql$execution$datasources$DefaultWriterContainer$$commitTask$1(WriterContainer.scala:275)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer$$anonfun$writeRows$1.apply$mcV$sp(WriterContainer.scala:257)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer$$anonfun$writeRows$1.apply(WriterContainer.scala:252)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer$$anonfun$writeRows$1.apply(WriterContainer.scala:252)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1348)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:258)
> ... 8 more
> Caused by: java.io.IOException: Failed to rename
> S3AFileStatus{path=s3a://foo/_temporary/0/_temporary/attempt_201703081855_0018_m_000966_0/part-r-00966-615ed714-58c1-4b89-be56-e47966737c75.snappy.parquet;
> isDirectory=false; length=111225342; replication=1; blocksize=33554432;
> modification_time=1488999342000; access_time=0; owner=; group=;
> permission=rw-rw-rw-; isSymlink=false} to
> s3a://foo/part-r-00966-615ed714-58c1-4b89-be56-e47966737c75.snappy.parquet
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:415)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:428)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:539)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:502)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:76)
> at
> org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitTask(WriterContainer.scala:211)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer.org$apache$spark$sql$execution$datasources$DefaultWriterContainer$$commitTask$1(WriterContainer.scala:270)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15191) Add Private/Unstable BulkDelete operations to supporting object stores for DistCP
[
https://issues.apache.org/jira/browse/HADOOP-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-15191:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> Add Private/Unstable BulkDelete operations to supporting object stores for
> DistCP
> -
>
> Key: HADOOP-15191
> URL: https://issues.apache.org/jira/browse/HADOOP-15191
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15191-001.patch, HADOOP-15191-002.patch,
> HADOOP-15191-003.patch, HADOOP-15191-004.patch
>
>
> Large scale DistCP with the -delete option doesn't finish in a viable time
> because of the final CopyCommitter doing a 1 by 1 delete of all missing
> files. This isn't randomized (the list is sorted), and it's throttled by AWS.
> If bulk deletion of files was exposed as an API, distCP would do 1/1000 of
> the REST calls, so not get throttled.
> Proposed: add an initially private/unstable interface for stores,
> {{BulkDelete}} which declares a page size and offers a
> {{bulkDelete(List)}} operation for the bulk deletion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15210) Handle FNFE from S3Guard.getMetadataStore() in S3A initialize()
[
https://issues.apache.org/jira/browse/HADOOP-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-15210:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> Handle FNFE from S3Guard.getMetadataStore() in S3A initialize()
> ---
>
> Key: HADOOP-15210
> URL: https://issues.apache.org/jira/browse/HADOOP-15210
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Minor
>
> {{S3Guard.getMetadataStore()}} throws FileNotFoundExceptions up, as the
> comments say " rely on callers to catch and treat specially"
> S3A Filesystem doesn't do that, instead it will just fail
> FileSystem.initialize; the FNFE is generated by DynamoDBMetadataStore.
> Are we happy with this?
> Downgrading has some appeal: if you don't have the table, it will keep going.
> But failures could be a sign of bad config, so maybe silent recovery is bad.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13664) S3AInputStream to use a retry policy on read failures
[
https://issues.apache.org/jira/browse/HADOOP-13664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-13664:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> S3AInputStream to use a retry policy on read failures
> -
>
> Key: HADOOP-13664
> URL: https://issues.apache.org/jira/browse/HADOOP-13664
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> {{S3AInputStream}} has some retry logic to handle failures on a read: log and
> retry. We should move this over to a (possibly hard coded RetryPolicy with
> some sleep logic, so that longer-than-just-transient read failures can be
> handled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13967) S3ABlockOutputStream to support plugin point for different multipart strategies
[
https://issues.apache.org/jira/browse/HADOOP-13967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-13967:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> S3ABlockOutputStream to support plugin point for different multipart
> strategies
> ---
>
> Key: HADOOP-13967
> URL: https://issues.apache.org/jira/browse/HADOOP-13967
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> For 0-rename commits, we need to delay the final commit of a multipart PUT,
> instead saving the data needed to build that commit into the s3 bucket.
> This means changes to {{S3ABlockOutputStream}} so that it can support
> different policies on how to do this, "classic" and "delayed commit".
> Having this self contained means we can test it in isolation of anything else.
> I'm ignoring the old output stream...we will switch to fast output whenever a
> special destination path is encountered
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14800) eliminate double stack trace on some s3guard CLI failures
[
https://issues.apache.org/jira/browse/HADOOP-14800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14800:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> eliminate double stack trace on some s3guard CLI failures
> -
>
> Key: HADOOP-14800
> URL: https://issues.apache.org/jira/browse/HADOOP-14800
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
>
> {{s3guard destroy}] when there's no bucket ends up double-listing the stack
> trace, which is somewhat confusing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14530) Translate AWS SSE-KMS missing key exception to something
[ https://issues.apache.org/jira/browse/HADOOP-14530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-14530: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > Translate AWS SSE-KMS missing key exception to something > > > Key: HADOOP-14530 > URL: https://issues.apache.org/jira/browse/HADOOP-14530 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > when you use SSE-KMS and the ARN is invalid for that region, you get a 400 > bad request exception + special error text "KMS.NotFoundException".This could > be a special exception -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13973) S3A GET/HEAD requests failing: java.lang.IllegalStateException: Connection is not open/Connection pool shut down
[ https://issues.apache.org/jira/browse/HADOOP-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-13973: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > S3A GET/HEAD requests failing: java.lang.IllegalStateException: Connection is > not open/Connection pool shut down > > > Key: HADOOP-13973 > URL: https://issues.apache.org/jira/browse/HADOOP-13973 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: EC2 cluster >Reporter: Rajesh Balamohan >Assignee: Steve Loughran >Priority: Major > > S3 requests failing with an error coming from Http client, > "java.lang.IllegalStateException: Connection is not open" > Some online discussion implies that this is related to shared connection pool > shutdown & fixed in http client 4.4+. Hadoop & AWS SDK use v 4.5.2 so the fix > is in, we just need to make sure the pool is being set up right. > There's a problem here of course: it may require moving to a later version of > the AWS SDK, with the consequences on jackson , as seen in HADOOP-13050. > And that's if there is a patched version out there -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14620) S3A authentication failure for regions other than us-east-1
[
https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14620:
-
Fix Version/s: (was: 3.1.0)
Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix /
Duplicate / Cannot Reproduce.
> S3A authentication failure for regions other than us-east-1
> ---
>
> Key: HADOOP-14620
> URL: https://issues.apache.org/jira/browse/HADOOP-14620
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Ilya Fourmanov
>Priority: Minor
> Attachments: s3-403.txt
>
>
> hadoop fs s3a:// operations fail authentication for s3 buckets hosted in
> regions other than default us-east-1
> Steps to reproduce:
> # create s3 bucket in eu-west-1
> # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run
> following command:
> {code}
> hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls
> s3a://your-eu-west-1-hosted-bucket/
> {code}
> Expected behaviour:
> You will see listing of the bucket
> Actual behaviour:
> You will get 403 Authentication Denied response for AWS S3.
> Reason is mismatch in string to sign as defined in
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html
> provided by hadoop and expected by AWS.
> If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes
> returned by AWS, you will see that AWS expects CanonicalizedResource to be in
> form
> /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/.
> Hadoop provides it as /your-eu-west-1-hosted-bucket/
> Note that AWS documentation doesn't explicitly state that endpoint or full
> dns address should be appended to CanonicalizedResource however practice
> shows it is actually required.
> I've also submitted this to AWS for them to correct behaviour or
> documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13969) S3A to support commit(path) operation, which commits all pending put commits in a path
[ https://issues.apache.org/jira/browse/HADOOP-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-13969: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > S3A to support commit(path) operation, which commits all pending put commits > in a path > -- > > Key: HADOOP-13969 > URL: https://issues.apache.org/jira/browse/HADOOP-13969 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > as well as creating and saving data with a pending-commit, s3a needs to add > the actual commit operation. > this would scan a directory, take its pending commits, read them in and > execute them. > issue: what to do on failures? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13059) S3a over-reacts to potentially transient network problems in its init() logic
[ https://issues.apache.org/jira/browse/HADOOP-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-13059: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > S3a over-reacts to potentially transient network problems in its init() logic > - > > Key: HADOOP-13059 > URL: https://issues.apache.org/jira/browse/HADOOP-13059 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13059-001.patch > > > If there's a reason for s3a not being able to connect to AWS, then the > constructor fails, even if this is a potentially transient event. > This happens because the code to check for a bucket existing will relay the > exceptions. > The constructor should catch IOEs against the remote FS, downgrade to warn > and let the code continue; it may fail later, but it may also recover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-15003: - Fix Version/s: (was: 3.1.0) Removing 3.1.0 fix-version from all JIRAs which are Invalid / Won't Fix / Duplicate / Cannot Reproduce. > Merge S3A committers into trunk: Yetus patch checker > > > Key: HADOOP-15003 > URL: https://issues.apache.org/jira/browse/HADOOP-15003 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, > HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, > HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, > HADOOP-13786-049.patch, HADOOP-13786-050.patch, HADOOP-13786-051.patch, > HADOOP-13786-052.patch, HADOOP-13786-053.patch, HADOOP-15033-testfix-1.diff > > > This is a Yetus only JIRA created to have Yetus review the > HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR > [https://github.com/apache/hadoop/pull/282] is stopping this happening in > HADOOP-14971. > Reviews should go into the PR/other task -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: (was: HADOOP.15331.001.patch)
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP-15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: HADOOP-15331.001.patch
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP-15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Commented] (HADOOP-15074) SequenceFile#Writer flush does not update the length of the written file.
[
https://issues.apache.org/jira/browse/HADOOP-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408715#comment-16408715
]
Arpit Agarwal commented on HADOOP-15074:
Hi Shashikant, I don't think we should change the default behavior of
{{DFSOutputStream#hsync(void)}}. There is a
{{DFSOutputStream#hsync(EnumSet syncFlags)}} overload that optionally
takes sync flags.
If the desired behavior is for {{Writer#hsync}} to update the length on the
NameNode, then it should pass the UPDATE_LENGTH flag. i.e. make the change in
SequenceFileWriter.
> SequenceFile#Writer flush does not update the length of the written file.
> -
>
> Key: HADOOP-15074
> URL: https://issues.apache.org/jira/browse/HADOOP-15074
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
>
> SequenceFile#Writer flush does not update the length of the file. This
> happens because as part of the flush, {{UPDATE_LENGTH}} flag is not passed to
> the DFSOutputStream#hsync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408698#comment-16408698
]
Rushabh S Shah commented on HADOOP-14445:
-
One more high level comment.
Lets say we are running in an environment where the job submitter and task
client are running with the new code but RM is not upgraded yet.
Job submitter will have {{KMS_DELEGATION_TOKEN}} and {{kms-dt}} tokens in its
credentials bag.
Lets say the job ran for more than 24 hours so that RM has to renew the token.
Since the RM is old, it will only renew {{kms-dt}} token and _will fail to
renew KMS_DELEGATION_TOKEN_.
Now when the new task tries to select the token via {{getKMSToken}}, it will
always select the token with kind {{KMS_DELEGATION_TOKEN}}.
If I understand the DelegationToken correctly, even if it fails to renew token
specific to \{{KMS_DELEGATION_TOKEN}}, that token will be valid since
\{{kms-dt}} is a copy of \{{KMS_D_T}}.
Lets just write a test case to verify.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408643#comment-16408643 ] Bharat Viswanadham commented on HADOOP-15334: - +1 LGTM. > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408624#comment-16408624 ] Arpit Agarwal edited comment on HADOOP-15334 at 3/21/18 9:41 PM: - v01 patch upgrades the surefire plugin to v2.21.0 (release notes [here|https://blogs.apache.org/maven/entry/apache-maven-surefire-plugin-version]) was (Author: arpitagarwal): Patch upgrades the surefire plugin to v2.21.0 (release notes [here|https://blogs.apache.org/maven/entry/apache-maven-surefire-plugin-version]) > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408624#comment-16408624 ] Arpit Agarwal commented on HADOOP-15334: Patch upgrades the surefire plugin to v2.21.0 (release notes [here|https://blogs.apache.org/maven/entry/apache-maven-surefire-plugin-version]) > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15334: --- Description: Recent versions of the surefire plugin suppress summary test execution output in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). (was: Recent versions of the surefire plugin suppress summary test execution output in quiet mode. This was recently fixed in plugin version 2.21.0 (via SUREFIRE-1436).) > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This is now fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15334: --- Attachment: HADOOP-15334.01.patch > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This was recently fixed in plugin version 2.21.0 (via > SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15334) Upgrade Maven surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15334: --- Status: Patch Available (was: Open) > Upgrade Maven surefire plugin > - > > Key: HADOOP-15334 > URL: https://issues.apache.org/jira/browse/HADOOP-15334 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Major > Attachments: HADOOP-15334.01.patch > > > Recent versions of the surefire plugin suppress summary test execution output > in quiet mode. This was recently fixed in plugin version 2.21.0 (via > SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15334) Upgrade Maven surefire plugin
Arpit Agarwal created HADOOP-15334: -- Summary: Upgrade Maven surefire plugin Key: HADOOP-15334 URL: https://issues.apache.org/jira/browse/HADOOP-15334 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Recent versions of the surefire plugin suppress summary test execution output in quiet mode. This was recently fixed in plugin version 2.21.0 (via SUREFIRE-1436). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408601#comment-16408601
]
Miklos Szegedi commented on HADOOP-15331:
-
Fixing checkstyle.
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP.15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: HADOOP.15331.001.patch
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch, HADOOP.15331.001.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408444#comment-16408444
]
genericqa commented on HADOOP-15331:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 244 unchanged - 0 fixed = 245 total (was 244) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 36s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 32s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915514/HADOOP-15331.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux bc7316535487 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6c63cc7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14364/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14364/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14364/testReport/ |
| Max. process+thread count | 1500 (vs. ulimit of 1) |
[jira] [Commented] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting
[
https://issues.apache.org/jira/browse/HADOOP-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408427#comment-16408427
]
genericqa commented on HADOOP-14855:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
3s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 21s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-14855 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915526/HADOOP-14855.002.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux bfafd6281812 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6c63cc7 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14365/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14365/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Hadoop scripts may errantly believe a daemon is still running, preventing it
> from starting
> --
>
> Key: HADOOP-14855
> URL: https://issues.apache.org/jira/browse/HADOOP-14855
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha4
>Reporter: Aaron T. Myers
>Assignee: Robert Kanter
>Priority: Major
> Attachments: HADOOP-14855.001.patch, HADOOP-14855.002.patch
>
>
> I encountered a case recently where the NN wouldn't start, with the error
> message "namenode is running as process 16769. Stop it first." In fact the
> NN was not running at all, but rather another long-running process was
> running with this pid.
> It looks to me like our scripts just check to see if _any_ process is running
> with the pid that the NN (or any Hadoop daemon) most recently ran with. This
> is clearly not a fool-proof way of checking to see if a particular type of
> daemon is now
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408390#comment-16408390
]
genericqa commented on HADOOP-15331:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 244 unchanged - 0 fixed = 245 total (was 244) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 10s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 19s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915514/HADOOP-15331.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c7459610bd56 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6c63cc7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14363/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14363/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14363/testReport/ |
| Max.
[jira] [Updated] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting
[ https://issues.apache.org/jira/browse/HADOOP-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-14855: --- Attachment: HADOOP-14855.002.patch > Hadoop scripts may errantly believe a daemon is still running, preventing it > from starting > -- > > Key: HADOOP-14855 > URL: https://issues.apache.org/jira/browse/HADOOP-14855 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha4 >Reporter: Aaron T. Myers >Assignee: Robert Kanter >Priority: Major > Attachments: HADOOP-14855.001.patch, HADOOP-14855.002.patch > > > I encountered a case recently where the NN wouldn't start, with the error > message "namenode is running as process 16769. Stop it first." In fact the > NN was not running at all, but rather another long-running process was > running with this pid. > It looks to me like our scripts just check to see if _any_ process is running > with the pid that the NN (or any Hadoop daemon) most recently ran with. This > is clearly not a fool-proof way of checking to see if a particular type of > daemon is now running, as some other process could start running with the > same pid since the daemon in question was previously shut down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting
[ https://issues.apache.org/jira/browse/HADOOP-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408294#comment-16408294 ] Robert Kanter commented on HADOOP-14855: The 002 patch fixes the new shellcheck issues (I had an old version installed before). > Hadoop scripts may errantly believe a daemon is still running, preventing it > from starting > -- > > Key: HADOOP-14855 > URL: https://issues.apache.org/jira/browse/HADOOP-14855 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha4 >Reporter: Aaron T. Myers >Assignee: Robert Kanter >Priority: Major > Attachments: HADOOP-14855.001.patch, HADOOP-14855.002.patch > > > I encountered a case recently where the NN wouldn't start, with the error > message "namenode is running as process 16769. Stop it first." In fact the > NN was not running at all, but rather another long-running process was > running with this pid. > It looks to me like our scripts just check to see if _any_ process is running > with the pid that the NN (or any Hadoop daemon) most recently ran with. This > is clearly not a fool-proof way of checking to see if a particular type of > daemon is now running, as some other process could start running with the > same pid since the daemon in question was previously shut down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15333) Deprecate or remove Groups#getUserToGroupsMappingServiceWithLoadedConfiguration
[
https://issues.apache.org/jira/browse/HADOOP-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15333:
---
Summary: Deprecate or remove
Groups#getUserToGroupsMappingServiceWithLoadedConfiguration (was: Depreciate
or remove Groups#getUserToGroupsMappingServiceWithLoadedConfiguration)
> Deprecate or remove
> Groups#getUserToGroupsMappingServiceWithLoadedConfiguration
> ---
>
> Key: HADOOP-15333
> URL: https://issues.apache.org/jira/browse/HADOOP-15333
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
>
> Groups#getUserToGroupsMappingServiceWithLoadedConfiguration introduced in
> YARN-1676 reinitializes its static instance i.e \{{GROUPS}}. This instance
> is stored by UGI for local reference. Use of
> getUserToGroupsMappingServiceWithLoadedConfiguration results in UGI reference
> detached from Parent class. This jira intends to discuss probable removal or
> depreciation of this function to remove innocuous bug resulting from it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15333) Depreciate or remove Groups#getUserToGroupsMappingServiceWithLoadedConfiguration
[
https://issues.apache.org/jira/browse/HADOOP-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408202#comment-16408202
]
Ajay Kumar commented on HADOOP-15333:
-
ping [~xyao], [~ste...@apache.org], [~arpitagarwal] for any thoughts around
this.
> Depreciate or remove
> Groups#getUserToGroupsMappingServiceWithLoadedConfiguration
>
>
> Key: HADOOP-15333
> URL: https://issues.apache.org/jira/browse/HADOOP-15333
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
>
> Groups#getUserToGroupsMappingServiceWithLoadedConfiguration introduced in
> YARN-1676 reinitializes its static instance i.e \{{GROUPS}}. This instance
> is stored by UGI for local reference. Use of
> getUserToGroupsMappingServiceWithLoadedConfiguration results in UGI reference
> detached from Parent class. This jira intends to discuss probable removal or
> depreciation of this function to remove innocuous bug resulting from it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15333) Depreciate or remove Groups#getUserToGroupsMappingServiceWithLoadedConfiguration
Ajay Kumar created HADOOP-15333:
---
Summary: Depreciate or remove
Groups#getUserToGroupsMappingServiceWithLoadedConfiguration
Key: HADOOP-15333
URL: https://issues.apache.org/jira/browse/HADOOP-15333
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ajay Kumar
Assignee: Ajay Kumar
Groups#getUserToGroupsMappingServiceWithLoadedConfiguration introduced in
YARN-1676 reinitializes its static instance i.e \{{GROUPS}}. This instance is
stored by UGI for local reference. Use of
getUserToGroupsMappingServiceWithLoadedConfiguration results in UGI reference
detached from Parent class. This jira intends to discuss probable removal or
depreciation of this function to remove innocuous bug resulting from it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408158#comment-16408158
]
Gergo Repas commented on HADOOP-15331:
--
+1 (non-binding)
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Status: Patch Available (was: Open)
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0, 2.7.5, 2.8.3, 2.9.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Updated] (HADOOP-15331) Race condition causing org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream
[
https://issues.apache.org/jira/browse/HADOOP-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated HADOOP-15331:
Attachment: HADOOP-15331.000.patch
> Race condition causing org.apache.hadoop.conf.Configuration: error parsing
> conf java.io.BufferedInputStream
> ---
>
> Key: HADOOP-15331
> URL: https://issues.apache.org/jira/browse/HADOOP-15331
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.10.0
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: HADOOP-15331.000.patch
>
>
> There is a race condition in the way Hadoop handles the Configuration class.
> The scenario is the following. Let's assume that there are two threads
> sharing the same Configuration class. One adds some resources to the
> configuration, while the other one clones it. Resources are loaded lazily in
> a deferred call to {{loadResources()}}. If the cloning happens after adding
> the resources but before parsing them, some temporary resources like input
> stream pointers are cloned. Eventually both copies will load the input stream
> resources pointing to the same input streams. One parses the input stream XML
> and closes it updating it's own copy of the resource. The other one has
> another pointer to the same input stream. When it tries to load it, it will
> crash with a stream closed exception.
> Here is an example unit test:
> {code:java}
> @Test
> public void testResourceRace() {
> InputStream is =
> new BufferedInputStream(new ByteArrayInputStream(
> "".getBytes()));
> Configuration conf = new Configuration();
> // Thread 1
> conf.addResource(is);
> // Thread 2
> Configuration confClone = new Configuration(conf);
> // Thread 2
> confClone.get("firstParse");
> // Thread 1
> conf.get("secondParse");
> }{code}
> Example real world stack traces:
> {code:java}
> 2018-02-28 08:23:19,589 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1420)
> at
> org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:161)
> at
> org.apache.hadoop.ipc.Server.refreshServiceAclWithLoadedConfiguration(Server.java:607)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshServiceAcls(AdminService.java:586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:188)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1231)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1421)
> {code}
> Another example:
> {code:java}
> 2018-02-28 08:23:20,702 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf java.io.BufferedInputStream@7741d346
> com.ctc.wstx.exc.WstxIOException: Stream closed
> at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:578)
> at
> com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2853)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2817)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1326)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1298)
> at
>
[jira] [Comment Edited] (HADOOP-11423) [Umbrella] Support Java 10 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407915#comment-16407915 ] Lars Francke edited comment on HADOOP-11423 at 3/21/18 1:41 PM: Java 10 has been released. But I'm strongly in favor of only targeting the LTS releases. The next of which will be Java 11 (18.9) in September 2018. The "current" LTS release (even though it's not called that) is Java 8. http://www.oracle.com/technetwork/java/eol-135779.html was (Author: lars_francke): Java 10 has been released. But I'm strongly in favor of only targeting the LTS releases. The next of which will be Java 11 in September 2018. > [Umbrella] Support Java 10 in Hadoop > > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13939) hadoop jar -libjars not working with mainClass argument
[
https://issues.apache.org/jira/browse/HADOOP-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407917#comment-16407917
]
Mark Wood commented on HADOOP-13939:
This applies to any generic option, not just -libjars; perhaps the title should
be updated appropriately? I just spent too much time trying to figure out why
my generic options weren't getting recognized!
> hadoop jar -libjars not working with mainClass argument
> ---
>
> Key: HADOOP-13939
> URL: https://issues.apache.org/jira/browse/HADOOP-13939
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Valentin Logvinskiy
>Priority: Major
>
> If we have
> {code:java}
> public class MainClass extends Configured implements Tool {
> public static void main(String[] args) throws Exception {
> ToolRunner.run(conf, new MainClass(), args);
> }
>
> ...
>
>@Override
> public int run(String[] args) throws Exception {
> ...
> }
> }
> {code}
> Running this code with -libjars argument in distributed mode will not work.
> {noformat}
> hadoop jar file.jar MainClass -libjars /path/to/file
> {noformat}
> This not working due to error in GenericOptionsParser
> GenericOptionsParser.java:
> {code}
> commandLine = parser.parse(opts, args, true);
> {code}
> As far as 3rd argument is "true" this mean:
> {noformat}
> * @param stopAtNonOption specifies whether to continue parsing the
> * arguments if a non option is encountered.
> {noformat}
> Our run arguments are:
> {noformat}
> MainClass -libjars /path/to/file
> {noformat}
> MainClass is nonOption and after this argument everything else don't start
> parsing.
> If we have only arguments then processGeneralOptions method will return
> nothing and will not add tmpjars config.
> Changing to
> {noformat}
> commandLine = parser.parse(opts, args, false);
> {noformat}
> works for me, but could affect a lot of developers (in theory)
> Fast fix - do not use mainClass argument at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11423) [Umbrella] Support Java 10 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407915#comment-16407915 ] Lars Francke commented on HADOOP-11423: --- Java 10 has been released. But I'm strongly in favor of only targeting the LTS releases. The next of which will be Java 11 in September 2018. > [Umbrella] Support Java 10 in Hadoop > > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407522#comment-16407522
]
genericqa commented on HADOOP-15253:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 39s{color} | {color:orange} root: The patch generated 2 new + 207 unchanged
- 0 fixed = 209 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 25s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}213m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15253 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915419/HADOOP-15253.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d3705d8def55 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 69fe440 |
