[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952432#comment-16952432 ] lqjacklee commented on HADOOP-15870: HADOOP-15870-006.patch provide the option `support-available-is-zero` to check the available is zero support provide the option `support-available-is-positive` to check the available is positive support provide the option `support-available-at-eof` to check the available at eof support > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch, HADOOP-15870-006.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15870: --- Attachment: HADOOP-15870-006.patch > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch, HADOOP-15870-006.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950806#comment-16950806 ] lqjacklee commented on HADOOP-15870: [~ayushtkn] Thank you very much. We will check it. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949859#comment-16949859
]
lqjacklee commented on HADOOP-15961:
[~ste...@apache.org] Thanks the reply , I will create another patch for this
task.
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch,
> HADOOP-15961-003.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16577) Build fails as can't retrieve websocket-servlet
[
https://issues.apache.org/jira/browse/HADOOP-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931457#comment-16931457
]
lqjacklee commented on HADOOP-16577:
Retrieval of
/org/eclipse/jetty/websocket/websocket-server/9.3.27.v20190418/websocket-server-9.3.27.v20190418.jar
from M2Repository(id=snapshots) is forbidden by repository policy SNAPSHOT.
It seems that the policy settings issue.
> Build fails as can't retrieve websocket-servlet
> ---
>
> Key: HADOOP-16577
> URL: https://issues.apache.org/jira/browse/HADOOP-16577
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Erkin Alp Güney
>Priority: Blocker
> Labels: dependencies
>
> I encountered this error when building Hadoop:
> Downloading:
> https://repository.apache.org/content/repositories/snapshots/org/eclipse/jetty/websocket/websocket-server/9.3.27.v20190418/websocket-server-9.3.27.v20190418.jar
> Sep 15, 2019 7:54:39 AM
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec
> execute
> INFO: I/O exception
> (org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException)
> caught when processing request to {s}->https://repository.apache.org:443: The
> target server failed to respond
> Sep 15, 2019 7:54:39 AM
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec
> execute
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16543) Cached DNS name resolution error
[ https://issues.apache.org/jira/browse/HADOOP-16543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927182#comment-16927182 ] lqjacklee commented on HADOOP-16543: [~roliu][HDDS-1933|https://issues.apache.org/jira/browse/HDDS-1933] should see the same issue ? > Cached DNS name resolution error > > > Key: HADOOP-16543 > URL: https://issues.apache.org/jira/browse/HADOOP-16543 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Roger Liu >Priority: Major > > In Kubernetes, the a node may go down and then come back later with a > different IP address. Yarn clients which are already running will be unable > to rediscover the node after it comes back up due to caching the original IP > address. This is problematic for cases such as Spark HA on Kubernetes, as the > node containing the resource manager may go down and come back up, meaning > existing node managers must then also be restarted. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16543) Cached DNS name resolution error
[ https://issues.apache.org/jira/browse/HADOOP-16543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921957#comment-16921957 ] lqjacklee commented on HADOOP-16543: I think something need to be changed. 1, should change IP to alias/dns name 2, should update the alias/dns name to the new IP address ( this part should be done in the k8s). > Cached DNS name resolution error > > > Key: HADOOP-16543 > URL: https://issues.apache.org/jira/browse/HADOOP-16543 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Roger Liu >Priority: Major > > In Kubernetes, the a node may go down and then come back later with a > different IP address. Yarn clients which are already running will be unable > to rediscover the node after it comes back up due to caching the original IP > address. This is problematic for cases such as Spark HA on Kubernetes, as the > node containing the resource manager may go down and come back up, meaning > existing node managers must then also be restarted. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892330#comment-16892330 ] lqjacklee commented on HADOOP-16435: [~kgyrtkirk] I mean that the session is end is not identity that the server is end . > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891915#comment-16891915 ] lqjacklee commented on HADOOP-16435: [~kgyrtkirk] org.apache.hadoop.ipc.metrics.RpcMetrics#shutdown will be called when the server is stopped. so I wonder whether the solution is ok? > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 0
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876225#comment-16876225
]
lqjacklee commented on HADOOP-15847:
[~ste...@apache.org] Thanks to the feedback. Have updated the merge request,
please review, thanks.
> S3Guard testConcurrentTableCreations to set r & w capacity == 0
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch, HADOOP-15847-002.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Status: In Progress (was: Patch Available)
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch, HADOOP-15847-002.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875538#comment-16875538
]
lqjacklee commented on HADOOP-15847:
https://github.com/apache/hadoop/pull/1037
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch, HADOOP-15847-002.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875536#comment-16875536
]
lqjacklee commented on HADOOP-15847:
[~ste...@apache.org] how to check whether the bucket is on-demand?
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch, HADOOP-15847-002.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3A committers for zero-rename commits to S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874307#comment-16874307 ] lqjacklee commented on HADOOP-13786: [~ste...@apache.org] great job. > Add S3A committers for zero-rename commits to S3 endpoints > -- > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch, > HADOOP-13786-038.patch, HADOOP-13786-039.patch, > HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch, > HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch, > HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch, > HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch, > HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch, > HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch, > HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch, > HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch, > HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch, > HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch, > HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch, > HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch, > HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch, > HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch, > HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch, > HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch, > MAPREDUCE-6823-003.patch, cloud-intergration-test-failure.log, > objectstore.pdf, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16399) Add missing getFileBlockLocations overload to FilterFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874216#comment-16874216
]
lqjacklee commented on HADOOP-16399:
[~ste...@apache.org] which subclass is not implemented?
> Add missing getFileBlockLocations overload to FilterFileSystem
> --
>
> Key: HADOOP-16399
> URL: https://issues.apache.org/jira/browse/HADOOP-16399
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.2.0
>Reporter: David Phillips
>Priority: Major
>
> The {{getFileBlockLocations(Path, long, long)}} overload is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15410) hoop-auth org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider org.apache.log4j package compile error
[ https://issues.apache.org/jira/browse/HADOOP-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15410: --- Resolution: Fixed Status: Resolved (was: Patch Available) > hoop-auth > org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider > org.apache.log4j package compile error > -- > > Key: HADOOP-15410 > URL: https://issues.apache.org/jira/browse/HADOOP-15410 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: lqjack >Priority: Major > Attachments: HADOOP-15410.000.patch > > > When run the > org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider , > IDE will automatic compile the java class , but unlucky org.apache.log4j > compile failed. > should change the pom.xml > > log4j > log4j > runtime > > to > > log4j > log4j > compile > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15410) hoop-auth org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider org.apache.log4j package compile error
[ https://issues.apache.org/jira/browse/HADOOP-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-15410: -- Assignee: lqjacklee > hoop-auth > org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider > org.apache.log4j package compile error > -- > > Key: HADOOP-15410 > URL: https://issues.apache.org/jira/browse/HADOOP-15410 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: lqjack >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15410.000.patch > > > When run the > org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider , > IDE will automatic compile the java class , but unlucky org.apache.log4j > compile failed. > should change the pom.xml > > log4j > log4j > runtime > > to > > log4j > log4j > compile > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15961:
---
Attachment: HADOOP-15961-003.patch
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch,
> HADOOP-15961-003.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809420#comment-16809420
]
lqjacklee commented on HADOOP-15961:
[~ste...@apache.org] I am working to this task, thanks.
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16121) Cannot build in dev docker environment
[
https://issues.apache.org/jira/browse/HADOOP-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee resolved HADOOP-16121.
Resolution: Resolved
Assignee: lqjacklee
> Cannot build in dev docker environment
> --
>
> Key: HADOOP-16121
> URL: https://issues.apache.org/jira/browse/HADOOP-16121
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.0
> Environment: Darwin lqjacklee-MacBook-Pro.local 18.2.0 Darwin Kernel
> Version 18.2.0: Mon Nov 12 20:24:46 PST 2018;
> root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
>Reporter: lqjacklee
>Assignee: lqjacklee
>Priority: Minor
>
> Operation as below :
>
> 1, run the docker daemon
> 2, run ./start-build-env.sh
> 3, mvn clean package -DskipTests
>
> Response from the command line :
>
> [ERROR] Plugin org.apache.maven.plugins:maven-surefire-plugin:2.17 or one of
> its dependencies could not be resolved: Failed to read artifact descriptor
> for org.apache.maven.plugins:maven-surefire-plugin:jar:2.17: Could not
> transfer artifact org.apache.maven.plugins:maven-surefire-plugin:pom:2.17
> from/to central (https://repo.maven.apache.org/maven2):
> /home/liu/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.pom.part.lock
> (No such file or directory) -> [Help 1]
>
> solution :
> a, sudo chmod -R 775 ${USER_HOME}/.m2/
> b, sudo chown -R ${USER_NAME} ${USER_HOME}/.m2
>
> After try the way , it still in trouble.
>
> c, sudo mvn clean package -DskipTests. but in this way, will download the
> file (pom, jar ) duplicated ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16121) Cannot build in dev docker environment
[
https://issues.apache.org/jira/browse/HADOOP-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789128#comment-16789128
]
lqjacklee commented on HADOOP-16121:
sudo ln -s /opt/protobuf/bin/protoc /usr/bin/protoc
sudo ln -s /opt/cmake/bin/cmake /usr/bin/cmake
> Cannot build in dev docker environment
> --
>
> Key: HADOOP-16121
> URL: https://issues.apache.org/jira/browse/HADOOP-16121
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.0
> Environment: Darwin lqjacklee-MacBook-Pro.local 18.2.0 Darwin Kernel
> Version 18.2.0: Mon Nov 12 20:24:46 PST 2018;
> root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
>Reporter: lqjacklee
>Priority: Minor
>
> Operation as below :
>
> 1, run the docker daemon
> 2, run ./start-build-env.sh
> 3, mvn clean package -DskipTests
>
> Response from the command line :
>
> [ERROR] Plugin org.apache.maven.plugins:maven-surefire-plugin:2.17 or one of
> its dependencies could not be resolved: Failed to read artifact descriptor
> for org.apache.maven.plugins:maven-surefire-plugin:jar:2.17: Could not
> transfer artifact org.apache.maven.plugins:maven-surefire-plugin:pom:2.17
> from/to central (https://repo.maven.apache.org/maven2):
> /home/liu/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.pom.part.lock
> (No such file or directory) -> [Help 1]
>
> solution :
> a, sudo chmod -R 775 ${USER_HOME}/.m2/
> b, sudo chown -R ${USER_NAME} ${USER_HOME}/.m2
>
> After try the way , it still in trouble.
>
> c, sudo mvn clean package -DskipTests. but in this way, will download the
> file (pom, jar ) duplicated ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16123) Lack of protoc in docker
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-16123: --- Attachment: HADOOP-16123-002.patch > Lack of protoc in docker > > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: lqjacklee >Priority: Minor > Attachments: HADOOP-16123-001.patch, HADOOP-16123-002.patch > > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16123) Lack of protoc in docker
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788575#comment-16788575 ] lqjacklee commented on HADOOP-16123: 002 patch remove the test case from the yarn nodenamage > Lack of protoc in docker > > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: lqjacklee >Priority: Minor > Attachments: HADOOP-16123-001.patch, HADOOP-16123-002.patch > > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16123) Lack of protoc in docker
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-16123: --- Status: Patch Available (was: Open) > Lack of protoc in docker > > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: lqjacklee >Priority: Minor > Attachments: HADOOP-16123-001.patch > > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16175) DynamoDBLocal Support
lqjacklee created HADOOP-16175: -- Summary: DynamoDBLocal Support Key: HADOOP-16175 URL: https://issues.apache.org/jira/browse/HADOOP-16175 Project: Hadoop Common Issue Type: New Feature Components: common Reporter: lqjacklee Assignee: lqjacklee DynamoDB Local ([https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html)] provide the function that the user/developer can run the local environment without depending on AWS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16123) Lack of protoc in docker
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-16123: --- Attachment: HADOOP-16123-001.patch > Lack of protoc in docker > > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: lqjacklee >Priority: Minor > Attachments: HADOOP-16123-001.patch > > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16123) Lack of protoc in docker
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788516#comment-16788516 ] lqjacklee commented on HADOOP-16123: After change the code : hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt . #add_executable(test-oom-listener # main/native/oom-listener/impl/oom_listener.c # main/native/oom-listener/impl/oom_listener.h # main/native/oom-listener/test/oom_listener_test_main.cc #) #target_link_libraries(test-oom-listener gtest rt) #output_directory(test-oom-listener test) It can work fine now. I wonder whether the test can be removed ? > Lack of protoc in docker > > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: lqjacklee >Priority: Minor > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15920: --- Attachment: HADOOP-15920-07.patch > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch, > HADOOP-15870-003.patch, HADOOP-15870-004.patch, HADOOP-15870-005.patch, > HADOOP-15920-06.patch, HADOOP-15920-07.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773615#comment-16773615 ] lqjacklee commented on HADOOP-15920: HADOOP-15920-06.patch fix checkstyle > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch, > HADOOP-15870-003.patch, HADOOP-15870-004.patch, HADOOP-15870-005.patch, > HADOOP-15920-06.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15920: --- Attachment: HADOOP-15920-06.patch > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch, > HADOOP-15870-003.patch, HADOOP-15870-004.patch, HADOOP-15870-005.patch, > HADOOP-15920-06.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16123) Lack of protoc in docker
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-16123: -- Assignee: (was: lqjacklee) > Lack of protoc in docker > > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: lqjacklee >Priority: Minor > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773579#comment-16773579 ] lqjacklee commented on HADOOP-15920: Thanks, I will format it. > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch, > HADOOP-15870-003.patch, HADOOP-15870-004.patch, HADOOP-15870-005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16123) Lack of protoc
[ https://issues.apache.org/jira/browse/HADOOP-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772513#comment-16772513 ] lqjacklee commented on HADOOP-16123: [~ste...@apache.org] please help check that, thanks . > Lack of protoc > --- > > Key: HADOOP-16123 > URL: https://issues.apache.org/jira/browse/HADOOP-16123 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: lqjacklee >Assignee: Steve Loughran >Priority: Minor > > During build the source code , do the steps as below : > > 1, run docker daemon > 2, ./start-build-env.sh > 3, sudo mvn clean install -DskipTests -Pnative > the response prompt that : > [ERROR] Failed to execute goal > org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) > on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: > 'protoc --version' did not return a version -> > [Help 1] > However , when execute the command : whereis protoc > liu@a65d187055f9:~/hadoop$ whereis protoc > protoc: /opt/protobuf/bin/protoc > > the PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin > > liu@a65d187055f9:~/hadoop$ protoc --version > libprotoc 2.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16123) Lack of protoc
lqjacklee created HADOOP-16123: -- Summary: Lack of protoc Key: HADOOP-16123 URL: https://issues.apache.org/jira/browse/HADOOP-16123 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.3.0 Reporter: lqjacklee Assignee: Steve Loughran During build the source code , do the steps as below : 1, run docker daemon 2, ./start-build-env.sh 3, sudo mvn clean install -DskipTests -Pnative the response prompt that : [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1] However , when execute the command : whereis protoc liu@a65d187055f9:~/hadoop$ whereis protoc protoc: /opt/protobuf/bin/protoc the PATH value : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cmake/bin:/opt/protobuf/bin liu@a65d187055f9:~/hadoop$ protoc --version libprotoc 2.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16121) Cannot build in dev docker environment
lqjacklee created HADOOP-16121:
--
Summary: Cannot build in dev docker environment
Key: HADOOP-16121
URL: https://issues.apache.org/jira/browse/HADOOP-16121
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.3.0
Environment: Darwin lqjacklee-MacBook-Pro.local 18.2.0 Darwin Kernel
Version 18.2.0: Mon Nov 12 20:24:46 PST 2018;
root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
Reporter: lqjacklee
Assignee: Steve Loughran
Operation as below :
1, run the docker daemon
2, run ./start-build-env.sh
3, mvn clean package -DskipTests
Response from the command line :
[ERROR] Plugin org.apache.maven.plugins:maven-surefire-plugin:2.17 or one of
its dependencies could not be resolved: Failed to read artifact descriptor for
org.apache.maven.plugins:maven-surefire-plugin:jar:2.17: Could not transfer
artifact org.apache.maven.plugins:maven-surefire-plugin:pom:2.17 from/to
central (https://repo.maven.apache.org/maven2):
/home/liu/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.17/maven-surefire-plugin-2.17.pom.part.lock
(No such file or directory) -> [Help 1]
solution :
a, sudo chmod -R 775 ${USER_HOME}/.m2/
b, sudo chown -R ${USER_NAME} ${USER_HOME}/.m2
After try the way , it still in trouble.
c, sudo mvn clean package -DskipTests. but in this way, will download the file
(pom, jar ) duplicated ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771474#comment-16771474
]
lqjacklee commented on HADOOP-15961:
[~ste...@apache.org] I will update the code today. thanks .
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15961:
---
Attachment: HADOOP-15961-002.patch
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747620#comment-16747620
]
lqjacklee edited comment on HADOOP-15961 at 1/21/19 2:47 AM:
-
[~ste...@apache.org] Which Branch should I apply ?
was (Author: jack-lee):
[~ste...@apache.org] Which Branch I should I apply ?
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747620#comment-16747620
]
lqjacklee commented on HADOOP-15961:
[~ste...@apache.org] Which Branch I should I apply ?
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746123#comment-16746123
]
lqjacklee commented on HADOOP-15961:
[~ste...@apache.org]Thanks the comment.
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744588#comment-16744588 ] lqjacklee commented on HADOOP-15920: [~ste...@apache.org] Thanks the comment , I will update. > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch, > HADOOP-15870-003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744586#comment-16744586 ] lqjacklee commented on HADOOP-15990: [~ste...@apache.org] Got it , I will double check again. > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744585#comment-16744585
]
lqjacklee commented on HADOOP-15961:
I configure the region : ap-southeast-1
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742645#comment-16742645
]
lqjacklee commented on HADOOP-15961:
I notice that test pass , do you get any issue about the patch ?
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742642#comment-16742642 ] lqjacklee commented on HADOOP-15990: [~ste...@apache.org] What's error do you occur ? Do I need update new patch? > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Status: Patch Available (was: Reopened) > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740284#comment-16740284 ] lqjacklee commented on HADOOP-15994: [~ajisakaa] sorry to update the status. > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738097#comment-16738097 ] lqjacklee commented on HADOOP-15994: [https://github.com/apache/hadoop/pull/460] update the Jackson version to 2.9.8 > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch, HADOOP-15994-002.patch, > HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737934#comment-16737934 ] lqjacklee commented on HADOOP-15994: [~ajisakaa] do we need update the version so frequency ? > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch, HADOOP-15994-002.patch, > HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16016) TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
[
https://issues.apache.org/jira/browse/HADOOP-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732564#comment-16732564
]
lqjacklee commented on HADOOP-16016:
I notice that the enabled cipher suits is 43, while after building the SSL ,
the supported cipher suits is 59. So I wonder whether it is just the reason
you referred ?
> TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
> ---
>
> Key: HADOOP-16016
> URL: https://issues.apache.org/jira/browse/HADOOP-16016
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
> Environment: Java 1.8.0_191 or upper
>Reporter: Jason Lowe
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16016-002.patch, HADOOP-16016.01.patch
>
>
> I have seen a couple of precommit builds across JIRAs fail in
> TestSSLFactory#testServerWeakCiphers with the error:
> {noformat}
> [ERROR] TestSSLFactory.testServerWeakCiphers:240 Expected to find 'no
> cipher suites in common' but got unexpected
> exception:javax.net.ssl.SSLHandshakeException: No appropriate protocol
> (protocol is disabled or cipher suites are inappropriate)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16021) SequenceFile.createWriter appendIfExists codec cause NullPointerException
[
https://issues.apache.org/jira/browse/HADOOP-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731579#comment-16731579
]
lqjacklee commented on HADOOP-16021:
[~xinkenny] Thanks ,I will try to reproduce it .
> SequenceFile.createWriter appendIfExists codec cause NullPointerException
> -
>
> Key: HADOOP-16021
> URL: https://issues.apache.org/jira/browse/HADOOP-16021
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.3
> Environment: windows10 or Linux-centos , hadoop2.7.3, jdk8
>Reporter: asin
>Priority: Major
> Labels: bug
> Attachments: 055.png, 62.png, CompressionType.BLOCK-Not
> supported-error log.txt, CompressionType.NONE-NullPointerException-error
> log.txt
>
>
>
> I want append the data in a file , when i use SequenceFile.appendIfExists ,
> it throw NullPointerException at at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1119)
> when i remove the 'appendIfExists', it works, but it will cover old file.
>
> when i try use CompressionType.RECORD or CompressionType.BLOCK throw "not
> support" exception
>
> {code:java}
> // my code
> SequenceFile.Writer writer = null;
> writer = SequenceFile.createWriter(conf,
> SequenceFile.Writer.file(path),
> SequenceFile.Writer.keyClass(Text.class),
> SequenceFile.Writer.valueClass(Text.class),
> SequenceFile.Writer.appendIfExists(true) );
> {code}
>
> {code:java}
> // all my code
> public class Writer1 implements VoidFunction String>>> {
> private static Configuration conf = new Configuration();
> private int MAX_LINE = 3; // little num,for test
> @Override
> public void call(Iterator> iterator) throws
> Exception {
> int partitionId = TaskContext.get().partitionId();
> int count = 0;
> SequenceFile.Writer writer = null;
> while (iterator.hasNext()) {
> Tuple2 tp = iterator.next();
> Path path = new Path("D:/tmp-doc/logs/logs.txt");
> if (writer == null)
> writer = SequenceFile.createWriter(conf,
> SequenceFile.Writer.file(path),
> SequenceFile.Writer.keyClass(Text.class),
> SequenceFile.Writer.valueClass(Text.class),
> SequenceFile.Writer.appendIfExists(true)
> );
> writer.append(new Text(tp._1), new Text(tp._2));
> count++;
> if (count > MAX_LINE) {
> IOUtils.closeStream(writer);
> count = 0;
> writer = SequenceFile.createWriter(... // same as above
> }
> }
> if (count > 0) {
> IOUtils.closeStream(writer);
> }
> IOUtils.closeStream(writer);
> }
> }
> {code}
> // above code call by below
> {code:java}
> import com.xxx.algo.hadoop.Writer1
> import com.xxx.algo.utils.Utils
> import kafka.serializer.StringDecoder
> import org.apache.spark.sql.SparkSession
> import org.apache.spark.streaming.kafka.KafkaUtils
> import org.apache.spark.streaming.{Durations, StreamingContext}
> import org.apache.spark.{SparkConf, SparkContext}
> object KafkaSparkStreamingApp {
> def main(args: Array[String]): Unit = {
> val kafka = "192.168.30.4:9092,192.168.30.5:9092,192.168.30.6:9092"
> val zk = "192.168.30.4:2181,192.168.30.5:2181,192.168.30.6:2181"
> val topics = Set("test.aries.collection.appevent.biz")
> val tag = "biz"
> val durationSeconds = 5000
> val conf = new SparkConf()
> conf.setAppName("user-log-consumer")
> .set("spark.serilizer","org.apache.spark.serializer.KryoSerializer")
> .set("spark.kryo.registrationRequired", "true")
> .set("spark.defalut.parallelism","2")
> .set("spark.rdd.compress","true")
> .setMaster("local[2]")
> val sc = new SparkContext(conf)
> val session = SparkSession.builder()
> .config(conf)
> .getOrCreate()
> val ssc = new StreamingContext(sc,
> Durations.milliseconds(durationSeconds))
> val kafkaParams = Map[String, String](
> "metadata.broker.list" -> kafka,
> "bootstrap.servers" -> kafka,
> "zookeeper.connect" -> zk,
> "group.id" -> "recommend_stream_spark",
> "key.serializer" ->
> "org.apache.kafka.common.serialization.StringSerializer",
> "key.deserializer" ->
> "org.apache.kafka.common.serialization.StringDeserializer",
> "value.deserializer" ->
> "org.apache.kafka.common.serialization.StringDeserializer"
> )
> val stream = KafkaUtils.createDirectStream[String, String, StringDecoder,
> StringDecoder](
> ssc,
> kafkaParams,
> topics
> )
> val timeFieldName = "log_time"
>
[jira] [Comment Edited] (HADOOP-16016) TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
[
https://issues.apache.org/jira/browse/HADOOP-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731155#comment-16731155
]
lqjacklee edited comment on HADOOP-16016 at 12/31/18 4:05 AM:
--
Just ignore the exception is one solution. however the issue still be there, so
I wonder you need change the enabled cipher suites. However there exist the
security issue , we should just enable the ones (or all supported)? I want to
submit the patch for us to review and update the solution .thanks .
{code:java}
private SSLEngineResult wrap(SSLEngine engine, ByteBuffer from,
ByteBuffer to) throws Exception {
String[] supportedCipherSuites = engine.getSupportedCipherSuites();
engine.setEnabledCipherSuites(supportedCipherSuites);
SSLEngineResult result = engine.wrap(from, to);
runDelegatedTasks(result, engine);
return result;
}
{code}
was (Author: jack-lee):
Just ignore the exception is one solution. however the issue still be there, so
I wonder you need change the enabled cipher suites. However there exist the
security issue , we should just enable the ones (or all supported). I want to
submit the patch for us to review and update the solution .thanks .
{code:java}
private SSLEngineResult wrap(SSLEngine engine, ByteBuffer from,
ByteBuffer to) throws Exception {
String[] supportedCipherSuites = engine.getSupportedCipherSuites();
engine.setEnabledCipherSuites(supportedCipherSuites);
SSLEngineResult result = engine.wrap(from, to);
runDelegatedTasks(result, engine);
return result;
}
{code}
> TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
> ---
>
> Key: HADOOP-16016
> URL: https://issues.apache.org/jira/browse/HADOOP-16016
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
> Environment: Java 1.8.0_191 or upper
>Reporter: Jason Lowe
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16016-002.patch, HADOOP-16016.01.patch
>
>
> I have seen a couple of precommit builds across JIRAs fail in
> TestSSLFactory#testServerWeakCiphers with the error:
> {noformat}
> [ERROR] TestSSLFactory.testServerWeakCiphers:240 Expected to find 'no
> cipher suites in common' but got unexpected
> exception:javax.net.ssl.SSLHandshakeException: No appropriate protocol
> (protocol is disabled or cipher suites are inappropriate)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16016) TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
[
https://issues.apache.org/jira/browse/HADOOP-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731155#comment-16731155
]
lqjacklee commented on HADOOP-16016:
Just ignore the exception is one solution. however the issue still be there, so
I wonder you need change the enabled cipher suites. However there exist the
security issue , we should just enable the ones (or all supported). I want to
submit the patch for us to review and update the solution .thanks .
{code:java}
private SSLEngineResult wrap(SSLEngine engine, ByteBuffer from,
ByteBuffer to) throws Exception {
String[] supportedCipherSuites = engine.getSupportedCipherSuites();
engine.setEnabledCipherSuites(supportedCipherSuites);
SSLEngineResult result = engine.wrap(from, to);
runDelegatedTasks(result, engine);
return result;
}
{code}
> TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
> ---
>
> Key: HADOOP-16016
> URL: https://issues.apache.org/jira/browse/HADOOP-16016
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
> Environment: Java 1.8.0_191 or upper
>Reporter: Jason Lowe
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16016-002.patch, HADOOP-16016.01.patch
>
>
> I have seen a couple of precommit builds across JIRAs fail in
> TestSSLFactory#testServerWeakCiphers with the error:
> {noformat}
> [ERROR] TestSSLFactory.testServerWeakCiphers:240 Expected to find 'no
> cipher suites in common' but got unexpected
> exception:javax.net.ssl.SSLHandshakeException: No appropriate protocol
> (protocol is disabled or cipher suites are inappropriate)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16016) TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
[
https://issues.apache.org/jira/browse/HADOOP-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-16016:
---
Attachment: HADOOP-16016-002.patch
> TestSSLFactory#testServerWeakCiphers sporadically fails in precommit builds
> ---
>
> Key: HADOOP-16016
> URL: https://issues.apache.org/jira/browse/HADOOP-16016
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
> Environment: Java 1.8.0_191 or upper
>Reporter: Jason Lowe
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16016-002.patch, HADOOP-16016.01.patch
>
>
> I have seen a couple of precommit builds across JIRAs fail in
> TestSSLFactory#testServerWeakCiphers with the error:
> {noformat}
> [ERROR] TestSSLFactory.testServerWeakCiphers:240 Expected to find 'no
> cipher suites in common' but got unexpected
> exception:javax.net.ssl.SSLHandshakeException: No appropriate protocol
> (protocol is disabled or cipher suites are inappropriate)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16024) org.apache.hadoop.security.ssl.TestSSLFactory#testServerWeakCiphers failed
lqjacklee created HADOOP-16024: -- Summary: org.apache.hadoop.security.ssl.TestSSLFactory#testServerWeakCiphers failed Key: HADOOP-16024 URL: https://issues.apache.org/jira/browse/HADOOP-16024 Project: Hadoop Common Issue Type: Bug Components: common Reporter: lqjacklee The enabled cipher suites locally is : TLS_ECDHE_RSA_WITH_RC4_128_SHA. however I found it is excluded. The track tree : java.lang.AssertionError: Expected to find 'no cipher suites in common' but got unexpected exception: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at sun.security.ssl.Handshaker.activate(Handshaker.java:509) at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:714) at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1212) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.hadoop.security.ssl.TestSSLFactory.wrap(TestSSLFactory.java:248) at org.apache.hadoop.security.ssl.TestSSLFactory.testServerWeakCiphers(TestSSLFactory.java:220) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) at org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:350) at org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:327) at org.apache.hadoop.security.ssl.TestSSLFactory.testServerWeakCiphers(TestSSLFactory.java:240) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at
[jira] [Updated] (HADOOP-16024) org.apache.hadoop.security.ssl.TestSSLFactory#testServerWeakCiphers failed
[ https://issues.apache.org/jira/browse/HADOOP-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-16024: --- Description: The enabled cipher suites locally is : enabledProtocols = "[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]" enabledCipherSuites = [TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5]" . however I found it is excluded. The track tree : java.lang.AssertionError: Expected to find 'no cipher suites in common' but got unexpected exception: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at sun.security.ssl.Handshaker.activate(Handshaker.java:509) at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:714) at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1212) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.hadoop.security.ssl.TestSSLFactory.wrap(TestSSLFactory.java:248) at org.apache.hadoop.security.ssl.TestSSLFactory.testServerWeakCiphers(TestSSLFactory.java:220) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) at org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:350) at org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:327) at org.apache.hadoop.security.ssl.TestSSLFactory.testServerWeakCiphers(TestSSLFactory.java:240) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at
[jira] [Commented] (HADOOP-16021) SequenceFile.createWriter appendIfExists codec cause NullPointerException
[
https://issues.apache.org/jira/browse/HADOOP-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729639#comment-16729639
]
lqjacklee commented on HADOOP-16021:
[~xinkenny] could provide the whole code ? thanks .
> SequenceFile.createWriter appendIfExists codec cause NullPointerException
> -
>
> Key: HADOOP-16021
> URL: https://issues.apache.org/jira/browse/HADOOP-16021
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.3
> Environment: windows10, hadoop2.7.3, jdk8
>Reporter: asin
>Priority: Major
> Labels: bug
> Attachments: 57.png
>
>
>
> I want append the data in a file , when i use SequenceFile.appendIfExists ,
> it throw NullPointerException at at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:1119)
> when i remove the 'appendIfExists', it works, but it will cover old file.
>
> when i try use CompressionType.RECORD or CompressionType.BLOCK throw "not
> support" exception
>
> {code:java}
> // my code
> SequenceFile.Writer writer = null;
> writer = SequenceFile.createWriter(conf,
> SequenceFile.Writer.file(path),
> SequenceFile.Writer.keyClass(Text.class),
> SequenceFile.Writer.valueClass(Text.class),
> SequenceFile.Writer.appendIfExists(true) );
> {code}
>
>
> {{more info
> see:[https://stackoverflow.com/questions/53943978/hadoop-sequencefile-createwriter-appendifexists-codec-cause-nullpointerexception]}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16019) ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration
[ https://issues.apache.org/jira/browse/HADOOP-16019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-16019: --- Description: * when the config ZK_DTSM_ZK_KERBEROS_KEYTAB or ZK_DTSM_ZK_KERBEROS_PRINCIPAL are not set, the IllegalArgumentException message cannot be logged. (was: when the config ZK_DTSM_ZK_KERBEROS_KEYTAB or ZK_DTSM_ZK_KERBEROS_PRINCIPAL are not set, the IllegalArgumentException message cannot be logged.) > ZKDelegationTokenSecretManager won't log exception message occured in > function setJaasConfiguration > --- > > Key: HADOOP-16019 > URL: https://issues.apache.org/jira/browse/HADOOP-16019 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.1.0 >Reporter: luhuachao >Priority: Minor > Attachments: HADOOP-16019.1.patch > > > * when the config ZK_DTSM_ZK_KERBEROS_KEYTAB or > ZK_DTSM_ZK_KERBEROS_PRINCIPAL are not set, the IllegalArgumentException > message cannot be logged. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15991) testMultipartUpload timing out
[ https://issues.apache.org/jira/browse/HADOOP-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723479#comment-16723479 ] lqjacklee commented on HADOOP-15991: # what happens if you don't run with s3guard on? ## The configuration is off # what are the s3guard settings for that bucket (e.g IO allocation). ## the default one, just change the region and credentials # How far is that AWS region from you? ## PING s3.ap-south-1.amazonaws.com (52.219.66.29): 56 data bytes 64 bytes from 52.219.66.29: icmp_seq=0 ttl=32 time=513.278 ms # and what is your bandwidth, especially uploading ## DOWNLOAD 48.58 Mbps /UPLOAD 3.46 Mbps > testMultipartUpload timing out > -- > > Key: HADOOP-15991 > URL: https://issues.apache.org/jira/browse/HADOOP-15991 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.2.0 >Reporter: lqjacklee >Assignee: lqjacklee >Priority: Minor > > timeout of S3 mpu tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723468#comment-16723468
]
lqjacklee commented on HADOOP-15847:
[~gabor.bota] Thanks the comment. In order to reduce the cost in the test
case.We provide the option to limit .
I have change the logic only in the ITestS3GuardConcurrentOps. Please help
review. [^HADOOP-15847-002.patch]
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch, HADOOP-15847-002.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Attachment: HADOOP-15847-002.patch
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch, HADOOP-15847-002.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15991) testMultipartUpload timing out
[ https://issues.apache.org/jira/browse/HADOOP-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-15991: -- Assignee: lqjacklee > testMultipartUpload timing out > -- > > Key: HADOOP-15991 > URL: https://issues.apache.org/jira/browse/HADOOP-15991 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.2.0 >Reporter: lqjacklee >Assignee: lqjacklee >Priority: Minor > > timeout of S3 mpu tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15961:
---
Attachment: HADOOP-15961-001.patch
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Attachments: HADOOP-15961-001.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15961:
---
Status: Patch Available (was: Open)
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee reassigned HADOOP-15961:
--
Assignee: lqjacklee
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Attachment: HADOOP-15994-003.patch > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch, HADOOP-15994-002.patch, > HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Attachment: HADOOP-15994-002.patch > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch, HADOOP-15994-002.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Attachment: HADOOP-15994-001.patch > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Status: Patch Available (was: Open) > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-15994: -- Assignee: lqjacklee > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714238#comment-16714238 ] lqjacklee commented on HADOOP-15990: 1, provide the version to identity which version user configure. The default version is 1. > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15990: --- Attachment: HADOOP-15990-006.patch > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15990: --- Status: Patch Available (was: Open) > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15990 stopped by lqjacklee. -- > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15991) testMultipartUpload
lqjacklee created HADOOP-15991:
--
Summary: testMultipartUpload
Key: HADOOP-15991
URL: https://issues.apache.org/jira/browse/HADOOP-15991
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 3.2.0
Reporter: lqjacklee
2018-12-10 09:58:56,482 [Thread-746] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:setup(184)) - Test filesystem =
s3a://jack-testlambda implemented by S3AFileSystem\{uri=s3a://jack-testlambda,
workingDir=s3a://jack-testlambda/user/liuquan, inputPolicy=normal,
partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000,
readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647,
serverSideEncryptionAlgorithm='NONE',
blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@3111299f,
metastore=DynamoDBMetadataStore{region=ap-southeast-1, tableName=test-h},
authoritative=false, useListV1=false, magicCommitter=false,
boundedExecutor=BlockingThreadPoolExecutorService\{SemaphoredDelegatingExecutor{permitCount=25,
available=25, waiting=0}, activeCount=0},
unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@78c596aa[Running,
pool size = 10, active threads = 0, queued tasks = 0, completed tasks = 20],
credentials=AWSCredentialProviderList[refcount= 2:
[SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider@4a214309], statistics
\{18901220 bytes read, 18912956 bytes written, 580 read ops, 0 large read ops,
843 write ops}, metrics \{{Context=s3aFileSystem}
\{s3aFileSystemId=94fd09ed-8145-4d1e-b11b-678151785e0b}
\{bucket=jack-testlambda} \{stream_opened=28} \{stream_close_operations=28}
\{stream_closed=28} \{stream_aborted=0} \{stream_seek_operations=0}
\{stream_read_exceptions=0} \{stream_forward_seek_operations=0}
\{stream_backward_seek_operations=0} \{stream_bytes_skipped_on_seek=0}
\{stream_bytes_backwards_on_seek=0} \{stream_bytes_read=18901220}
\{stream_read_operations=2734} \{stream_read_fully_operations=0}
\{stream_read_operations_incomplete=2171} \{stream_bytes_read_in_close=0}
\{stream_bytes_discarded_in_abort=0} \{files_created=49} \{files_copied=20}
\{files_copied_bytes=9441684} \{files_deleted=79}
\{fake_directories_deleted=630} \{directories_created=98}
\{directories_deleted=19} \{ignored_errors=71} \{op_copy_from_local_file=0}
\{op_create=55} \{op_create_non_recursive=6} \{op_delete=74} \{op_exists=80}
\{op_get_file_checksum=8} \{op_get_file_status=923} \{op_glob_status=19}
\{op_is_directory=0} \{op_is_file=0} \{op_list_files=8}
\{op_list_located_status=0} \{op_list_status=51} \{op_mkdirs=114} \{op_open=28}
\{op_rename=20} \{object_copy_requests=0} \{object_delete_requests=198}
\{object_list_requests=337} \{object_continue_list_requests=0}
\{object_metadata_requests=560} \{object_multipart_aborted=0}
\{object_put_bytes=18904764} \{object_put_requests=147}
\{object_put_requests_completed=147} \{stream_write_failures=0}
\{stream_write_block_uploads=0} \{stream_write_block_uploads_committed=0}
\{stream_write_block_uploads_aborted=0} \{stream_write_total_time=0}
\{stream_write_total_data=18904764} \{committer_commits_created=0}
\{committer_commits_completed=0} \{committer_jobs_completed=0}
\{committer_jobs_failed=0} \{committer_tasks_completed=0}
\{committer_tasks_failed=0} \{committer_bytes_committed=0}
\{committer_bytes_uploaded=0} \{committer_commits_failed=0}
\{committer_commits_aborted=0} \{committer_commits_reverted=0}
\{committer_magic_files_created=0}
\{s3guard_metadatastore_put_path_request=166}
\{s3guard_metadatastore_initialization=1} \{s3guard_metadatastore_retry=0}
\{s3guard_metadatastore_throttled=0} \{store_io_throttled=0}
\{object_put_requests_active=0} \{object_put_bytes_pending=0}
\{stream_write_block_uploads_active=0} \{stream_write_block_uploads_pending=49}
\{stream_write_block_uploads_data_pending=0}
\{S3guard_metadatastore_put_path_latencyNumOps=1}
\{S3guard_metadatastore_put_path_latency50thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency75thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency90thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency95thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency99thPercentileLatency=427507288}
\{S3guard_metadatastore_throttle_rateNumEvents=0}
\{S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate75thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate95thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz)=0} }}
2018-12-10 10:01:50,127 [Thread-746] INFO s3a.ITestS3AContractMultipartUploader
(ITestS3AContractMultipartUploader.java:teardown(108)) -
[jira] [Updated] (HADOOP-15991) testMultipartUpload
[
https://issues.apache.org/jira/browse/HADOOP-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15991:
---
Description:
2018-12-10 09:58:56,482 [Thread-746] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:setup(184)) - Test filesystem =
s3a://jack-testlambda implemented by S3AFileSystem{uri=s3a://jack-testlambda,
workingDir=s3a://jack-testlambda/user/jack, inputPolicy=normal,
partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000,
readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647,
serverSideEncryptionAlgorithm='NONE',
blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@3111299f,
metastore=DynamoDBMetadataStore
{region=ap-southeast-1, tableName=test-h}
, authoritative=false, useListV1=false, magicCommitter=false,
boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor
{permitCount=25, available=25, waiting=0}
, activeCount=0},
unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@78c596aa[Running,
pool size = 10, active threads = 0, queued tasks = 0, completed tasks = 20],
credentials=AWSCredentialProviderList[refcount= 2:
[SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider@4a214309], statistics
\{18901220 bytes read, 18912956 bytes written, 580 read ops, 0 large read ops,
843 write ops}, metrics {
{Context=s3aFileSystem}
{s3aFileSystemId=94fd09ed-8145-4d1e-b11b-678151785e0b}
\{bucket=jack-testlambda} \{stream_opened=28} \{stream_close_operations=28}
\{stream_closed=28} \{stream_aborted=0} \{stream_seek_operations=0}
\{stream_read_exceptions=0} \{stream_forward_seek_operations=0}
\{stream_backward_seek_operations=0} \{stream_bytes_skipped_on_seek=0}
\{stream_bytes_backwards_on_seek=0} \{stream_bytes_read=18901220}
\{stream_read_operations=2734} \{stream_read_fully_operations=0}
\{stream_read_operations_incomplete=2171} \{stream_bytes_read_in_close=0}
\{stream_bytes_discarded_in_abort=0} \{files_created=49} \{files_copied=20}
\{files_copied_bytes=9441684} \{files_deleted=79}
\{fake_directories_deleted=630} \{directories_created=98}
\{directories_deleted=19} \{ignored_errors=71} \{op_copy_from_local_file=0}
\{op_create=55} \{op_create_non_recursive=6} \{op_delete=74} \{op_exists=80}
\{op_get_file_checksum=8} \{op_get_file_status=923} \{op_glob_status=19}
\{op_is_directory=0} \{op_is_file=0} \{op_list_files=8}
\{op_list_located_status=0} \{op_list_status=51} \{op_mkdirs=114} \{op_open=28}
\{op_rename=20} \{object_copy_requests=0} \{object_delete_requests=198}
\{object_list_requests=337} \{object_continue_list_requests=0}
\{object_metadata_requests=560} \{object_multipart_aborted=0}
\{object_put_bytes=18904764} \{object_put_requests=147}
\{object_put_requests_completed=147} \{stream_write_failures=0}
\{stream_write_block_uploads=0} \{stream_write_block_uploads_committed=0}
\{stream_write_block_uploads_aborted=0} \{stream_write_total_time=0}
\{stream_write_total_data=18904764} \{committer_commits_created=0}
\{committer_commits_completed=0} \{committer_jobs_completed=0}
\{committer_jobs_failed=0} \{committer_tasks_completed=0}
\{committer_tasks_failed=0} \{committer_bytes_committed=0}
\{committer_bytes_uploaded=0} \{committer_commits_failed=0}
\{committer_commits_aborted=0} \{committer_commits_reverted=0}
\{committer_magic_files_created=0}
\{s3guard_metadatastore_put_path_request=166}
\{s3guard_metadatastore_initialization=1} \{s3guard_metadatastore_retry=0}
\{s3guard_metadatastore_throttled=0} \{store_io_throttled=0}
\{object_put_requests_active=0} \{object_put_bytes_pending=0}
\{stream_write_block_uploads_active=0} \{stream_write_block_uploads_pending=49}
\{stream_write_block_uploads_data_pending=0}
\{S3guard_metadatastore_put_path_latencyNumOps=1}
\{S3guard_metadatastore_put_path_latency50thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency75thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency90thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency95thPercentileLatency=427507288}
\{S3guard_metadatastore_put_path_latency99thPercentileLatency=427507288}
\{S3guard_metadatastore_throttle_rateNumEvents=0}
\{S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate75thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate95thPercentileFrequency (Hz)=0}
\{S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz)=0} }}
2018-12-10 10:01:50,127 [Thread-746] INFO
s3a.ITestS3AContractMultipartUploader
(ITestS3AContractMultipartUploader.java:teardown(108)) - Teardown: aborting
outstanding uploads under s3a://jack-testlambda/test
2018-12-10 10:01:50,618 [Thread-746] INFO
s3a.ITestS3AContractMultipartUploader
[jira] [Updated] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15990: --- Status: In Progress (was: Patch Available) > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[ https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714189#comment-16714189 ] lqjacklee commented on HADOOP-15990: [~ste...@apache.org] I will do that. thanks. > S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2 > > > Key: HADOOP-15990 > URL: https://issues.apache.org/jira/browse/HADOOP-15990 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15409-005.patch > > > in S3AFileSystem.initialize(), we check for the bucket existing with > verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't > check for auth issues. > s3. doesBucketExistV2() does at least validate credentials, and should be > switched to. This will help things fail faster > See SPARK-24000 > (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to > test everything) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15894) getFileChecksum() needs to adopt S3Guard
[
https://issues.apache.org/jira/browse/HADOOP-15894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee reassigned HADOOP-15894:
--
Assignee: lqjacklee
> getFileChecksum() needs to adopt S3Guard
>
>
> Key: HADOOP-15894
> URL: https://issues.apache.org/jira/browse/HADOOP-15894
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
>
> Encountered a 404 failure in
> {{ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted}}; newly
> created file wasn't seen. Even with S3guard enabled, that method isn't doing
> anything to query the store for it existing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-15894) getFileChecksum() needs to adopt S3Guard
[
https://issues.apache.org/jira/browse/HADOOP-15894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-15894 started by lqjacklee.
--
> getFileChecksum() needs to adopt S3Guard
>
>
> Key: HADOOP-15894
> URL: https://issues.apache.org/jira/browse/HADOOP-15894
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
>
> Encountered a 404 failure in
> {{ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted}}; newly
> created file wasn't seen. Even with S3guard enabled, that method isn't doing
> anything to query the store for it existing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Status: Patch Available (was: Open)
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713620#comment-16713620
]
lqjacklee commented on HADOOP-15847:
[~ste...@apache.org] Please help review the patch , and how to resolve the
exception.
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Attachment: HADOOP-15847-001.patch
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Attachment: (was: HADOOP-15847-001.patch)
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Assignee: lqjacklee
Status: Patch Available (was: Open)
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15847-001.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Status: Open (was: Patch Available)
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713512#comment-16713512
]
lqjacklee commented on HADOOP-15847:
When set the capacity to 1, test output render as below :
2018-12-08 11:55:57,912 [pool-4-thread-22] INFO
s3guard.ITestDynamoDBMetadataStoreScale
(ITestDynamoDBMetadataStoreScale.java:lambda$execute$8(432)) - Operation [0]
raised a throttled exception
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Max retries during batch
write exceeded (2) for DynamoDB. This may be because the write threshold of
DynamoDB is set too low.: Throttling (Service: S3Guard; Status Code: 503; Error
Code: Throttling; Request ID: n/a)
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Max retries during batch
write exceeded (2) for DynamoDB. This may be because the write threshold of
DynamoDB is set too low.: Throttling (Service: S3Guard; Status Code: 503; Error
Code: Throttling; Request ID: n/a)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.retryBackoffOnBatchWrite(DynamoDBMetadataStore.java:800)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.processBatchWriteRequest(DynamoDBMetadataStore.java:759)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPut(DynamoDBMetadataStore.java:845)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.put(DynamoDBMetadataStore.java:837)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.put(DynamoDBMetadataStore.java:831)
at
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale.lambda$test_030_BatchedWrite$0(ITestDynamoDBMetadataStoreScale.java:237)
at
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale.lambda$execute$8(ITestDynamoDBMetadataStoreScale.java:428)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.AmazonServiceException: Throttling (Service: S3Guard;
Status Code: 503; Error Code: Throttling; Request ID: n/a)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.retryBackoffOnBatchWrite(DynamoDBMetadataStore.java:791)
... 11 more
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15847-001.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15847) S3Guard testConcurrentTableCreations to set r & w capacity == 1
[
https://issues.apache.org/jira/browse/HADOOP-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lqjacklee updated HADOOP-15847:
---
Attachment: HADOOP-15847-001.patch
> S3Guard testConcurrentTableCreations to set r & w capacity == 1
> ---
>
> Key: HADOOP-15847
> URL: https://issues.apache.org/jira/browse/HADOOP-15847
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15847-001.patch
>
>
> I just found a {{testConcurrentTableCreations}} DDB table lurking in a
> region, presumably from an interrupted test. Luckily
> test/resources/core-site.xml forces the r/w capacity to be 10, but it could
> still run up bills.
> Recommend
> * explicitly set capacity = 1 for the test
> * and add comments in the testing docs about keeping cost down.
> I think we may also want to make this a scale-only test, so it's run less
> often
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15985) LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops
[ https://issues.apache.org/jira/browse/HADOOP-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-15985: -- Assignee: lqjacklee > LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops > -- > > Key: HADOOP-15985 > URL: https://issues.apache.org/jira/browse/HADOOP-15985 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Roman Leventov >Assignee: lqjacklee >Priority: Minor > Attachments: HADOOP-15985-002.patch, HADOOP-15985-1.patch > > > In this line: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java#L391], > instead of checking if the platform is 32- or 64-bit, it should check if > Unsafe.ARRAY_OBJECT_INDEX_SCALE is 4 or 8. > The result is that on 64-bit platforms, when Compressed Oops are on, > LightWeightGSet is two times denser than it is configured to be. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15985) LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops
[ https://issues.apache.org/jira/browse/HADOOP-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712217#comment-16712217 ] lqjacklee commented on HADOOP-15985: [~kihwal] Thanks reminder, I submit the wrong path. [^HADOOP-15985-002.patch] > LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops > -- > > Key: HADOOP-15985 > URL: https://issues.apache.org/jira/browse/HADOOP-15985 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Roman Leventov >Priority: Minor > Attachments: HADOOP-15985-002.patch, HADOOP-15985-1.patch > > > In this line: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java#L391], > instead of checking if the platform is 32- or 64-bit, it should check if > Unsafe.ARRAY_OBJECT_INDEX_SCALE is 4 or 8. > The result is that on 64-bit platforms, when Compressed Oops are on, > LightWeightGSet is two times denser than it is configured to be. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15985) LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops
[ https://issues.apache.org/jira/browse/HADOOP-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15985: --- Attachment: HADOOP-15985-002.patch > LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops > -- > > Key: HADOOP-15985 > URL: https://issues.apache.org/jira/browse/HADOOP-15985 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Roman Leventov >Priority: Minor > Attachments: HADOOP-15985-002.patch, HADOOP-15985-1.patch > > > In this line: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java#L391], > instead of checking if the platform is 32- or 64-bit, it should check if > Unsafe.ARRAY_OBJECT_INDEX_SCALE is 4 or 8. > The result is that on 64-bit platforms, when Compressed Oops are on, > LightWeightGSet is two times denser than it is configured to be. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15870 started by lqjacklee. -- > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15985) LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops
[ https://issues.apache.org/jira/browse/HADOOP-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711527#comment-16711527 ] lqjacklee commented on HADOOP-15985: [~kihwal] instead of checking if the platform is 32- or 64-bit, it should check if Unsafe.ARRAY_OBJECT_INDEX_SCALE is 4 or 8. > LightWeightGSet.computeCapacity() doesn't correctly account for CompressedOops > -- > > Key: HADOOP-15985 > URL: https://issues.apache.org/jira/browse/HADOOP-15985 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Roman Leventov >Priority: Minor > Attachments: HADOOP-15985-1.patch > > > In this line: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java#L391], > instead of checking if the platform is 32- or 64-bit, it should check if > Unsafe.ARRAY_OBJECT_INDEX_SCALE is 4 or 8. > The result is that on 64-bit platforms, when Compressed Oops are on, > LightWeightGSet is two times denser than it is configured to be. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15870 stopped by lqjacklee. -- > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15870: --- Attachment: HADOOP-15870-003.patch > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711522#comment-16711522 ] lqjacklee commented on HADOOP-15870: [^HADOOP-15870-003.patch] > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711524#comment-16711524 ] lqjacklee commented on HADOOP-15920: [^HADOOP-15870-003.patch] > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15920: --- Attachment: HADOOP-15870-003.patch > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch, > HADOOP-15870-003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
