[jira] [Reopened] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs reopened HADOOP-16193: - I must have made a mistake in testing this. Re-reviewing it, it's not possible for this to work and indeed it fails when I test it again. When writing to S3, the 'winner' will be the last-started-upload-that-completed-successfully. In the case of MPU this start is at init time, not complete time. So when the transient file comes in and is deleted then the MPU becomes moot. In the case of a versioned bucket, then the transient file doesn't unroll the latest version to the MPU because all the 'latest' knowledge is based on init time, not complete time. I will revert the commit. > add extra S3A MPU test to see what happens if a file is created during the MPU > -- > > Key: HADOOP-16193 > URL: https://issues.apache.org/jira/browse/HADOOP-16193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.1.3 > > > Proposed extra test for the S3A MPU: if you create and then delete a file > while an MPU is in progress, when you finally complete the MPU the new data > is present. > This verifies that the other FS operations don't somehow cancel the > in-progress upload, and that eventual consistency brings the latest value out. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913256#comment-16913256 ] Ewan Higgs commented on HADOOP-16193: - Merged the github branch. Thanks [~ste...@apache.org] > add extra S3A MPU test to see what happens if a file is created during the MPU > -- > > Key: HADOOP-16193 > URL: https://issues.apache.org/jira/browse/HADOOP-16193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Proposed extra test for the S3A MPU: if you create and then delete a file > while an MPU is in progress, when you finally complete the MPU the new data > is present. > This verifies that the other FS operations don't somehow cancel the > in-progress upload, and that eventual consistency brings the latest value out. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-16193: Fix Version/s: 3.1.3 > add extra S3A MPU test to see what happens if a file is created during the MPU > -- > > Key: HADOOP-16193 > URL: https://issues.apache.org/jira/browse/HADOOP-16193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.1.3 > > > Proposed extra test for the S3A MPU: if you create and then delete a file > while an MPU is in progress, when you finally complete the MPU the new data > is present. > This verifies that the other FS operations don't somehow cancel the > in-progress upload, and that eventual consistency brings the latest value out. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-16193: Resolution: Fixed Status: Resolved (was: Patch Available) > add extra S3A MPU test to see what happens if a file is created during the MPU > -- > > Key: HADOOP-16193 > URL: https://issues.apache.org/jira/browse/HADOOP-16193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Proposed extra test for the S3A MPU: if you create and then delete a file > while an MPU is in progress, when you finally complete the MPU the new data > is present. > This verifies that the other FS operations don't somehow cancel the > in-progress upload, and that eventual consistency brings the latest value out. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695205#comment-16695205
]
Ewan Higgs commented on HADOOP-14693:
-
In various tests (e.g. TestECSchema) we have {{@Rule Timeout globalTimeout =
new Timeout(3000);}} which means the entire test and all child tests have a
timeout set. Junit5 doesn't have this (yet). Discussion is here:
https://github.com/junit-team/junit5/issues/80
I'm not sure if this is a blocker for anyone. If the tests behave well enough
then maybe we can remove the timeouts?
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15338) Support Java 11 LTS in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675043#comment-16675043 ] Ewan Higgs commented on HADOOP-15338: - We are probably also blocked by https://issues.apache.org/jira/browse/MCOMPILER-342 > Support Java 11 LTS in Hadoop > - > > Key: HADOOP-15338 > URL: https://issues.apache.org/jira/browse/HADOOP-15338 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > > Oracle JDK 8 will be EoL during January 2019, and RedHat will end support for > OpenJDK 8 in October 2020 ([https://access.redhat.com/articles/1299013]), so > we need to support Java 11 LTS at least before October 2020. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11123) Fix Java 9 incompatibilies in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674820#comment-16674820
]
Ewan Higgs commented on HADOOP-11123:
-
{quote}
mark them as duplicate/superceded by HADOOP-15838 and move the open ones
there.
{quote}
HADOOP-15838 is "Copy files from SFTP to HDFS using DistCp failed with error"
HADOOP-15338 is "Support Java 11 LTS in Hadoop" :)
> Fix Java 9 incompatibilies in Hadoop
>
>
> Key: HADOOP-11123
> URL: https://issues.apache.org/jira/browse/HADOOP-11123
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
> Environment: Java 9
>Reporter: Steve Loughran
>Priority: Critical
>
> JIRA to cover/track issues related to Hadoop on Java 9.
> Java 9 will have some significant changes —one of which is the removal of
> various {{com.sun}} classes. These removals need to be handled or Hadoop will
> not be able to run on a Java 9 JVM
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15826) @Retries annotation of putObject() call & uses wrong
[ https://issues.apache.org/jira/browse/HADOOP-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15826: Status: Patch Available (was: Open) > @Retries annotation of putObject() call & uses wrong > > > Key: HADOOP-15826 > URL: https://issues.apache.org/jira/browse/HADOOP-15826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-15826-001.patch, HADOOP-15826-002.patch > > > The retry annotations of the S3AFilesystem putObject call and its > writeOperationsHelper use aren't in sync with what the code does. > Fix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15826) @Retries annotation of putObject() call & uses wrong
[ https://issues.apache.org/jira/browse/HADOOP-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15826: Status: Open (was: Patch Available) > @Retries annotation of putObject() call & uses wrong > > > Key: HADOOP-15826 > URL: https://issues.apache.org/jira/browse/HADOOP-15826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-15826-001.patch, HADOOP-15826-002.patch > > > The retry annotations of the S3AFilesystem putObject call and its > writeOperationsHelper use aren't in sync with what the code does. > Fix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15826) @Retries annotation of putObject() call & uses wrong
[ https://issues.apache.org/jira/browse/HADOOP-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15826: Attachment: HADOOP-15826-002.patch > @Retries annotation of putObject() call & uses wrong > > > Key: HADOOP-15826 > URL: https://issues.apache.org/jira/browse/HADOOP-15826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-15826-001.patch, HADOOP-15826-002.patch > > > The retry annotations of the S3AFilesystem putObject call and its > writeOperationsHelper use aren't in sync with what the code does. > Fix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15826) @Retries annotation of putObject() call & uses wrong
[ https://issues.apache.org/jira/browse/HADOOP-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651762#comment-16651762 ] Ewan Higgs commented on HADOOP-15826: - 002 * aforementioned suggestion to tag revertCommit as OnceTranslated > @Retries annotation of putObject() call & uses wrong > > > Key: HADOOP-15826 > URL: https://issues.apache.org/jira/browse/HADOOP-15826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-15826-001.patch, HADOOP-15826-002.patch > > > The retry annotations of the S3AFilesystem putObject call and its > writeOperationsHelper use aren't in sync with what the code does. > Fix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15848) ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error
[
https://issues.apache.org/jira/browse/HADOOP-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651752#comment-16651752
]
Ewan Higgs commented on HADOOP-15848:
-
01
- Override base class impl and do nothing. (I tried tagging it with {{@Ignore}}
but it didn't really seem to work.
> ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error
> -
>
> Key: HADOOP-15848
> URL: https://issues.apache.org/jira/browse/HADOOP-15848
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HADOOP-15848.01.patch
>
>
> To reproduce failure: {{mvn verify -Dscale
> -Dit.test=org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> -Dtest=skip}} against {{eu-west-1}}.
> Test output:
> {noformat}
> [INFO] Running
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 59.301 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> [ERROR]
> testMultipartUploadEmptyPart(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader)
> Time elapsed: 0.75 s <<< ERROR!
> java.lang.IllegalArgumentException: partNumber must be between 1 and 1
> inclusive, but is 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15848) ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error
[
https://issues.apache.org/jira/browse/HADOOP-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15848:
Assignee: Ewan Higgs
Status: Patch Available (was: Open)
> ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error
> -
>
> Key: HADOOP-15848
> URL: https://issues.apache.org/jira/browse/HADOOP-15848
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HADOOP-15848.01.patch
>
>
> To reproduce failure: {{mvn verify -Dscale
> -Dit.test=org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> -Dtest=skip}} against {{eu-west-1}}.
> Test output:
> {noformat}
> [INFO] Running
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 59.301 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> [ERROR]
> testMultipartUploadEmptyPart(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader)
> Time elapsed: 0.75 s <<< ERROR!
> java.lang.IllegalArgumentException: partNumber must be between 1 and 1
> inclusive, but is 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15848) ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error
[
https://issues.apache.org/jira/browse/HADOOP-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15848:
Attachment: HADOOP-15848.01.patch
> ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error
> -
>
> Key: HADOOP-15848
> URL: https://issues.apache.org/jira/browse/HADOOP-15848
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15848.01.patch
>
>
> To reproduce failure: {{mvn verify -Dscale
> -Dit.test=org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> -Dtest=skip}} against {{eu-west-1}}.
> Test output:
> {noformat}
> [INFO] Running
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 59.301 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
> [ERROR]
> testMultipartUploadEmptyPart(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader)
> Time elapsed: 0.75 s <<< ERROR!
> java.lang.IllegalArgumentException: partNumber must be between 1 and 1
> inclusive, but is 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15826) @Retries annotation of putObject() call & uses wrong
[
https://issues.apache.org/jira/browse/HADOOP-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644648#comment-16644648
]
Ewan Higgs commented on HADOOP-15826:
-
Good catch. revertCommit should also be fixed to have {{@OnceTranslated}}:
{code}
@Retries.RetryTranslated
public void revertCommit(String destKey) throws IOException {
once("revert commit", destKey,
() -> {
Path destPath = owner.keyToQualifiedPath(destKey);
owner.deleteObjectAtPath(destPath,
destKey, true);
owner.maybeCreateFakeParentDirectory(destPath);
}
);
}
{code}
> @Retries annotation of putObject() call & uses wrong
>
>
> Key: HADOOP-15826
> URL: https://issues.apache.org/jira/browse/HADOOP-15826
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15826-001.patch
>
>
> The retry annotations of the S3AFilesystem putObject call and its
> writeOperationsHelper use aren't in sync with what the code does.
> Fix
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15756) [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622015#comment-16622015 ] Ewan Higgs commented on HADOOP-15756: - Merged. > [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement > -- > > Key: HADOOP-15756 > URL: https://issues.apache.org/jira/browse/HADOOP-15756 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15756.01.patch > > > In JDK10, sun.net.util.IPAddressUtil is encapsulated and not accessible from > unnamed modules. This issue is to remove the usage of IPAddressUtil. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15764) [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622016#comment-16622016 ] Ewan Higgs commented on HADOOP-15764: - Merged. > [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement > - > > Key: HADOOP-15764 > URL: https://issues.apache.org/jira/browse/HADOOP-15764 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15764.01.patch, HADOOP-15764.02.patch, > HADOOP-15764.03.patch, HADOOP-15764.04.patch > > > In JDK10, sun.net.dns.ResolverConfiguration is encapsulated and not > accessible from unnamed modules. This issue is to remove the usage of > ResolverConfiguration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15764) [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622013#comment-16622013 ] Ewan Higgs commented on HADOOP-15764: - +1 for this change. > [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement > - > > Key: HADOOP-15764 > URL: https://issues.apache.org/jira/browse/HADOOP-15764 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15764.01.patch, HADOOP-15764.02.patch, > HADOOP-15764.03.patch, HADOOP-15764.04.patch > > > In JDK10, sun.net.dns.ResolverConfiguration is encapsulated and not > accessible from unnamed modules. This issue is to remove the usage of > ResolverConfiguration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15764) [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622006#comment-16622006 ] Ewan Higgs commented on HADOOP-15764: - 05 * Rebase as HADOOP-15756 was merged. > [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement > - > > Key: HADOOP-15764 > URL: https://issues.apache.org/jira/browse/HADOOP-15764 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15764.01.patch, HADOOP-15764.02.patch, > HADOOP-15764.03.patch, HADOOP-15764.04.patch > > > In JDK10, sun.net.dns.ResolverConfiguration is encapsulated and not > accessible from unnamed modules. This issue is to remove the usage of > ResolverConfiguration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15764) [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15764: Attachment: HADOOP-15764.04.patch > [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement > - > > Key: HADOOP-15764 > URL: https://issues.apache.org/jira/browse/HADOOP-15764 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15764.01.patch, HADOOP-15764.02.patch, > HADOOP-15764.03.patch, HADOOP-15764.04.patch > > > In JDK10, sun.net.dns.ResolverConfiguration is encapsulated and not > accessible from unnamed modules. This issue is to remove the usage of > ResolverConfiguration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15764) [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15764: Status: Open (was: Patch Available) > [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement > - > > Key: HADOOP-15764 > URL: https://issues.apache.org/jira/browse/HADOOP-15764 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15764.01.patch, HADOOP-15764.02.patch, > HADOOP-15764.03.patch, HADOOP-15764.04.patch > > > In JDK10, sun.net.dns.ResolverConfiguration is encapsulated and not > accessible from unnamed modules. This issue is to remove the usage of > ResolverConfiguration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15764) [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15764: Status: Patch Available (was: Open) > [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement > - > > Key: HADOOP-15764 > URL: https://issues.apache.org/jira/browse/HADOOP-15764 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15764.01.patch, HADOOP-15764.02.patch, > HADOOP-15764.03.patch, HADOOP-15764.04.patch > > > In JDK10, sun.net.dns.ResolverConfiguration is encapsulated and not > accessible from unnamed modules. This issue is to remove the usage of > ResolverConfiguration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15756) [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement
[ https://issues.apache.org/jira/browse/HADOOP-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621969#comment-16621969 ] Ewan Higgs commented on HADOOP-15756: - As the ResolverConfiguration is in another ticket, this looks like a good simple change. +1 No new tests are needed as this is an API migration. > [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement > -- > > Key: HADOOP-15756 > URL: https://issues.apache.org/jira/browse/HADOOP-15756 > Project: Hadoop Common > Issue Type: Sub-task > Components: net, util >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-15756.01.patch > > > In JDK10, sun.net.util.IPAddressUtil is encapsulated and not accessible from > unnamed modules. This issue is to remove the usage of IPAddressUtil. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617497#comment-16617497
]
Ewan Higgs commented on HADOOP-11423:
-
{quote}
But I'm strongly in favor of only targeting the LTS releases. The next of which
will be Java 11 (18.9) in September 2018. The "current" LTS release (even
though it's not called that) is Java 8.\{quote}
AIUI OpenJDK is now the official Java implementation that we should target.
OpenJDK has no LTS releases.
> [Umbrella] Fix Java 10 incompatibilities in Hadoop
> --
>
> Key: HADOOP-11423
> URL: https://issues.apache.org/jira/browse/HADOOP-11423
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: sneaky
>Priority: Major
>
> Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly
> works with Java 10 is important for the Apache community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617497#comment-16617497
]
Ewan Higgs edited comment on HADOOP-11423 at 9/17/18 1:03 PM:
--
{quote}But I'm strongly in favor of only targeting the LTS releases. The next
of which will be Java 11 (18.9) in September 2018. The "current" LTS release
(even though it's not called that) is Java 8.{quote}
AIUI OpenJDK is now the official Java implementation that we should target.
OpenJDK has no LTS releases.
was (Author: ehiggs):
{quote}
But I'm strongly in favor of only targeting the LTS releases. The next of which
will be Java 11 (18.9) in September 2018. The "current" LTS release (even
though it's not called that) is Java 8.\{quote}
AIUI OpenJDK is now the official Java implementation that we should target.
OpenJDK has no LTS releases.
> [Umbrella] Fix Java 10 incompatibilities in Hadoop
> --
>
> Key: HADOOP-11423
> URL: https://issues.apache.org/jira/browse/HADOOP-11423
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: sneaky
>Priority: Major
>
> Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly
> works with Java 10 is important for the Apache community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15756) [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement
[
https://issues.apache.org/jira/browse/HADOOP-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617291#comment-16617291
]
Ewan Higgs commented on HADOOP-15756:
-
This is a good change. Should we take the opportunity to remove
{{sun.net.dns.ResolverConfiguration}} as well? {{sun.net.dns}} only exports to
{{java.security.jgss}} and {{jdk.naming.dns}}
https://github.com/dmlloyd/openjdk/blob/jdk/jdk10/src/java.base/share/classes/module-info.java#L219
> [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement
> --
>
> Key: HADOOP-15756
> URL: https://issues.apache.org/jira/browse/HADOOP-15756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net, util
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15756.01.patch
>
>
> In JDK10, sun.net.util.IPAddressUtil is encapsulated and not accessible from
> unnamed modules. This issue is to remove the usage of IPAddressUtil.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[
https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578316#comment-16578316
]
Ewan Higgs commented on HADOOP-14663:
-
006
- Rebased onto trunk at {{a13929ddcb3b90044350ae1c23a1150e8b4b975b}}.
[~aw], is there anything blocking this from going in? (YETUS-? INFRA-?)
> Switch to OpenClover
>
>
> Key: HADOOP-14663
> URL: https://issues.apache.org/jira/browse/HADOOP-14663
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, test
>Affects Versions: 3.0.0-beta1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Minor
> Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch,
> HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch,
> HADOOP-14663.05.patch, HADOOP-14663.06.patch
>
>
> Clover has gone open source. We should switch to it's replacement
> (OpenClover) so that more people can run code coverage tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Attachment: HADOOP-14663.06.patch > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch, HADOOP-14663.06.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Status: Patch Available (was: Open) > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch, HADOOP-14663.06.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Status: Open (was: Patch Available) > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch, HADOOP-14663.06.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[
https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578130#comment-16578130
]
Ewan Higgs commented on HADOOP-15642:
-
{quote}I like the idea of documenting the upgrade process itself - especially
since it requires testing across multiple "components"!{quote}
+1 to this idea.
> Update to latest/recent version of aws-sdk for Hadoop 3.2
> -
>
> Key: HADOOP-15642
> URL: https://issues.apache.org/jira/browse/HADOOP-15642
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen
> Shot 2018-07-30 at 14.11.22.png
>
>
> Move to a later version of the AWS SDK library for a different set of
> features and issues.
> proposed version: 1.11.375
> One thing which doesn't work on the one we ship with is the ability to create
> assumed role sessions >1h, as there's a check in the client lib for
> role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration
> checks to the far end.
> see:
> [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/]
> * assuming later versions will extend assumed role life, docs will need
> changing,
> * Adding a test in HADOOP-15583 which expects an error message if you ask for
> a duration of 3h; this should act as the test to see what happens.
> * think this time would be good to explicitly write down the SDK update
> process
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15645: Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15645-001.patch, HADOOP-15645-002.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578117#comment-16578117 ] Ewan Higgs commented on HADOOP-15645: - LGTM. Merged. > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15645-001.patch, HADOOP-15645-002.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[
https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577955#comment-16577955
]
Ewan Higgs commented on HADOOP-15645:
-
002
- Rebase onto current head ( {{3ac07b720b7839a7fe6c83f4ccfe319b6a892501}} )
> ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding
> to DDB
> ---
>
> Key: HADOOP-15645
> URL: https://issues.apache.org/jira/browse/HADOOP-15645
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15645-001.patch, HADOOP-15645-002.patch
>
>
> If your bucket has a per-bucket setting to use S3Guard, then the
> ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep
> into the metastore.
> The rawfs should use clearBucketOption to clear the metastore binding, so
> guarantee a real raw fs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15645: Attachment: HADOOP-15645-002.patch > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15645-001.patch, HADOOP-15645-002.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15645: Status: Patch Available (was: Open) > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15645-001.patch, HADOOP-15645-002.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15645: Status: Open (was: Patch Available) > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15645-001.patch, HADOOP-15645-002.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Resolution: Fixed
Fix Version/s: 3.2
Status: Resolved (was: Patch Available)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Fix For: 3.2
>
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576-008.patch, HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15667) FileSystemMultipartUploader should verify that UploadHandle has non-0 length
[
https://issues.apache.org/jira/browse/HADOOP-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576126#comment-16576126
]
Ewan Higgs commented on HADOOP-15667:
-
001
* Move checkUploadId to {{MultipartUploader}} and add tests to make sure the
failure is triggered.
Probable reviewers: [~ste...@apache.org], [~virajith], [~chris.douglas]
> FileSystemMultipartUploader should verify that UploadHandle has non-0 length
>
>
> Key: HADOOP-15667
> URL: https://issues.apache.org/jira/browse/HADOOP-15667
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HADOOP-15667.001.patch
>
>
> The S3AMultipartUploader has a good check on the length of the UploadHandle.
> This should be moved to MultipartUploader, made protected, and called in the
> various implementations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15667) FileSystemMultipartUploader should verify that UploadHandle has non-0 length
[ https://issues.apache.org/jira/browse/HADOOP-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15667: Status: Patch Available (was: Open) > FileSystemMultipartUploader should verify that UploadHandle has non-0 length > > > Key: HADOOP-15667 > URL: https://issues.apache.org/jira/browse/HADOOP-15667 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HADOOP-15667.001.patch > > > The S3AMultipartUploader has a good check on the length of the UploadHandle. > This should be moved to MultipartUploader, made protected, and called in the > various implementations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15667) FileSystemMultipartUploader should verify that UploadHandle has non-0 length
[ https://issues.apache.org/jira/browse/HADOOP-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-15667: Attachment: HADOOP-15667.001.patch > FileSystemMultipartUploader should verify that UploadHandle has non-0 length > > > Key: HADOOP-15667 > URL: https://issues.apache.org/jira/browse/HADOOP-15667 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HADOOP-15667.001.patch > > > The S3AMultipartUploader has a good check on the length of the UploadHandle. > This should be moved to MultipartUploader, made protected, and called in the > various implementations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15667) FileSystemMultipartUploader should verify that UploadHandle has non-0 length
Ewan Higgs created HADOOP-15667: --- Summary: FileSystemMultipartUploader should verify that UploadHandle has non-0 length Key: HADOOP-15667 URL: https://issues.apache.org/jira/browse/HADOOP-15667 Project: Hadoop Common Issue Type: Bug Reporter: Ewan Higgs Assignee: Ewan Higgs The S3AMultipartUploader has a good check on the length of the UploadHandle. This should be moved to MultipartUploader, made protected, and called in the various implementations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573090#comment-16573090
]
Ewan Higgs commented on HADOOP-15576:
-
Merged.
Thanks, [~ste...@apache.org], for your input and refinements.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576-008.patch, HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571368#comment-16571368
]
Ewan Higgs commented on HADOOP-15576:
-
{quote}I think this is good to go in, if you are happy with my changes{quote}
The changes look good. It was originally my patch so I'll let you apply it if
it's the same to you.
{quote} And there's no need for protobuf; we aren't worrying about wire compat
over time, are we?
{quote}
When are we not worried about wire compat over time? :)
{quote}But we do need that spec still, which I'd like before the actual apps
using this stuff come together{quote}
I'll take a crack at the spec now.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576-008.patch, HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570240#comment-16570240
]
Ewan Higgs edited comment on HADOOP-15576 at 8/6/18 2:05 PM:
-
{quote}legit regression; the rejection of 0-part MPUs is breaking a mock test.
{quote}
This is checked in {{testCompleteEmptyUpload}} that you've added.
Is there anything outstanding here? Just the question of whether we want to go
full protobuf to avoid Java Serialization?
was (Author: ehiggs):
{quote}legit regression; the rejection of 0-part MPUs is breaking a mock test.
{quote}
This is checked in {{testCompleteEmptyUpload}} that you've added. AIUI, S3
requires an MPU to have at least 1 part uploaded even if that part is 0 bytes.
Is there anything outstanding here? Just the question of whether we want to go
full protobuf to avoid Java Serialization?
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576-008.patch, HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570240#comment-16570240
]
Ewan Higgs commented on HADOOP-15576:
-
{quote}legit regression; the rejection of 0-part MPUs is breaking a mock test.
{quote}
This is checked in {{testCompleteEmptyUpload}} that you've added. AIUI, S3
requires an MPU to have at least 1 part uploaded even if that part is 0 bytes.
Is there anything outstanding here? Just the question of whether we want to go
full protobuf to avoid Java Serialization?
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576-008.patch, HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567945#comment-16567945
]
Ewan Higgs commented on HADOOP-15576:
-
Thanks. The changes mostly look very good.
{quote}The rule "1+ handle must have been uploaded" is new, but it stops the
MPU complete on S3 failing.{quote}
In the javadoc you say it must be >1; but it should be >=1. Or did you see this
failing with only a single part being written?
{quote}
S3A part handles marshall (header, len, etag); unmarshall validates header &
extracts len and etag. Unit tests for this. Uses java DataInputStream, nothing
fancy.
{quote}
Should we bite the bullet and make this protobuf? Or is adding a dep on
protobuf and adding a protoc step to hadoop-aws ott?
{quote}Big issue there: what would this mean for a distcp working this way? I'd
propose: 0-byte files get treated as special, or at least there's a requirement
for a 0-byte upload. Which, if supported, is something else to test for.{quote}
Agreed.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576.001.patch, HADOOP-15576.002.patch, HADOOP-15576.003.patch,
> HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565100#comment-16565100
]
Ewan Higgs commented on HADOOP-15576:
-
I wonder if I've misconfigured the test because they actually run 3 times for
me:
{code:java}
$ mvn install -DskipShade -Dmaven.javadoc.skip=true -Pdist,parallel-tests
-DtestsThreadCount=8 -Djava.awt.headless=true -DS3Guard -DAuth
-Dit.test=ITestS3AContractMultipartUploader -pl :hadoop-aws -Dtest=wibble
{code}
{code:java}
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test
(default-integration-test) @ hadoop-aws ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.541 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test
(sequential-integration-tests) @ hadoop-aws ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.987 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ hadoop-aws
---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.966 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
{code}
{quote}tests take a v long time.{quote}
Out of interest, how long do they take for you?
{quote}
I'm going to do a revision of the patch, so I'll see if I can identify the
problem. I'll do what I can about scaling the patch down to be viable, and make
it a -Dscale test only.
{quote}
Thanks
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563967#comment-16563967
]
Ewan Higgs edited comment on HADOOP-15576 at 7/31/18 4:50 PM:
--
004
- Add test for empty uploadID
- Make the part size configurable by FS implementation as S3A requires 5MB part
sizes (except the last part).
- Use md5sum to compare the file contents so we don't need to compare multiple
5MB blocks together if they were a string.
- Fix META-INF/service for S3AMultipartUploader$Factory.
- Removed test based on comparing fs.open(pathHandle) since S3a doesn't support
opening by path handle.
{quote}
yes. If it's java serialization then it needs to be looked at to make sure it
defends against malicios stuff.
{quote}
Would you prefer protobuf for this? It would be an extra generation step + new
dependency.
was (Author: ehiggs):
004
- Add test for empty uploadID
- Make the part size configurable by FS implementation as S3A requires 5MB part
sizes (except the last part).
- Use md5sum to compare the file contents so we don't need to compare multiple
5MB blocks together if they were a string.
- Fix META-INF/service for S3AMultipartUploader$Factory.
{quote}
yes. If it's java serialization then it needs to be looked at to make sure it
defends against malicios stuff.
{quote}
Would you prefer protobuf for this? It would be an extra generation step + new
dependency.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563967#comment-16563967
]
Ewan Higgs commented on HADOOP-15576:
-
004
- Add test for empty uploadID
- Make the part size configurable by FS implementation as S3A requires 5MB part
sizes (except the last part).
- Use md5sum to compare the file contents so we don't need to compare multiple
5MB blocks together if they were a string.
- Fix META-INF/service for S3AMultipartUploader$Factory.
{quote}
yes. If it's java serialization then it needs to be looked at to make sure it
defends against malicios stuff.
{quote}
Would you prefer protobuf for this? It would be an extra generation step + new
dependency.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Open (was: Patch Available)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Attachment: HADOOP-15576.004.patch
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Patch Available (was: Open)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559857#comment-16559857
]
Ewan Higgs commented on HADOOP-15576:
-
003
- Added Test - second complete throws IOException
- Added Test abort unknown upload is a no-op.
{quote}
initiate two MPUs to the same dest. What happens? (interesting q; outcome may
vary on FS)
{quote}
This is interesting and worthwhile to know what happens, but should the
contract be consistent across all storage systems? e.g. S3 has one semantic;
Azure may have another.
For the tests where you talk about unknown uploads, what do you mean here? The
UploadHandle is an opaque handle constructed by the initialize function. So if
we construct one ourselves (s.t. the MPU doesn't know about it), then are we
testing malicious intent? That's a bit weird since it's just wrapping calls to
the underlying FS so we're only exploiting a subset of behaviour in the
underlying FS.
Anyway, I think that 003 is the first one worth considering committing.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Attachment: HADOOP-15576.003.patch
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Patch Available (was: Open)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Open (was: Patch Available)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558465#comment-16558465
]
Ewan Higgs commented on HADOOP-15576:
-
002
- Move test(s) to fit into the Contract test framework.
- Give asserts meaningful messages for failures.
- wrap fs.open with try-with-resources.
- Use assertPathExists to make sure the paths exist.
- Use string comparison assertEquals for testMultipartUpload,
testMultipartUploadReverseOrder
- Add 'commit' after 'abort' in abort test to verify we throw IOException.
Still need to add Steve's suggested further tests.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Attachment: HADOOP-15576.002.patch
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Open (was: Patch Available)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555818#comment-16555818
]
Ewan Higgs edited comment on HADOOP-15576 at 7/25/18 3:17 PM:
--
Hi,
[~ste...@apache.org] thanks for taking a look at the original patch.
001
- Extended the tests in {{AbstractSystemMultipartUploaderTest.java}} to check
the existence of the files after writing (and non-existence of files when
aborting).
- Use {{WriteOperationHelper}} in S3A. Note that I use length of 0 since this
is not passed into the complete or any other MPU bit. It's not required for S3;
only the stats collection. [~ste...@apache.org], thoughts on what we want to do
here for file length?
I still need to extend the tests as requested.
was (Author: ehiggs):
Hi,
[~ste...@apache.org] thanks for taking a look at the original patch.
001
- Extended the tests in {{AbstractSystemMultipartUploaderTest.java}} to check
the existence of the files after writing (and non-existence of files when
aborting).
- Use {{WriteOperationHelper}} in S3A. Note that I use length of 0 since this
is not passed into the complete or any other MPU bit. It's not required for S3;
only the stats collection. [~ste...@apache.org], thoughts on what we want to do
here for file length?
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Patch Available (was: Open)
Hi,
[~ste...@apache.org] thanks for taking a look at the original patch.
001
- Extended the tests in {{AbstractSystemMultipartUploaderTest.java}} to check
the existence of the files after writing (and non-existence of files when
aborting).
- Use {{WriteOperationHelper}} in S3A. Note that I use length of 0 since this
is not passed into the complete or any other MPU bit. It's not required for S3;
only the stats collection. [~ste...@apache.org], thoughts on what we want to do
here for file length?
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Attachment: HADOOP-15576.001.patch
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15623) Compiling hadoop-azure fails with jdk10 (javax javascript)
Ewan Higgs created HADOOP-15623:
---
Summary: Compiling hadoop-azure fails with jdk10 (javax javascript)
Key: HADOOP-15623
URL: https://issues.apache.org/jira/browse/HADOOP-15623
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Ewan Higgs
{code}
$ java -version
java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)
{code}
{code}
$ mvn install -DskipShade -Dmaven.javadoc.skip=true -Djava.awt.headless=true
-DskipTests -rf :hadoop-azure
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(create-parallel-tests-dirs) on project hadoop-azure: An Ant BuildException has
occured: Unable to create javax script engine for javascript
[ERROR] around Ant part
[jira] [Commented] (HADOOP-15384) distcp numListstatusThreads option doesn't get to -delete scan
[
https://issues.apache.org/jira/browse/HADOOP-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536897#comment-16536897
]
Ewan Higgs commented on HADOOP-15384:
-
I've tested this using 1, 5, and 20 threads and we get the expected performance
improvement when constructing the source list.
+1
> distcp numListstatusThreads option doesn't get to -delete scan
> --
>
> Key: HADOOP-15384
> URL: https://issues.apache.org/jira/browse/HADOOP-15384
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: tools/distcp
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15384-001.patch
>
>
> The distcp {{numListstatusThreads}} option isn't used when configuring the
> GlobbedCopyListing used in {{CopyComitter.deleteMissing()}}
> This means that for large scans of object stores, performance is
> significantly worse.
> Fix: pass the option down from the task conf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15384) distcp numListstatusThreads option doesn't get to -delete scan
[
https://issues.apache.org/jira/browse/HADOOP-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529875#comment-16529875
]
Ewan Higgs commented on HADOOP-15384:
-
The code LGTM. Trying to test this to see if there is a significant performance
impact on S3.
> distcp numListstatusThreads option doesn't get to -delete scan
> --
>
> Key: HADOOP-15384
> URL: https://issues.apache.org/jira/browse/HADOOP-15384
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: tools/distcp
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15384-001.patch
>
>
> The distcp {{numListstatusThreads}} option isn't used when configuring the
> GlobbedCopyListing used in {{CopyComitter.deleteMissing()}}
> This means that for large scans of object stores, performance is
> significantly worse.
> Fix: pass the option down from the task conf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15445) TestCryptoAdminCLI test failure when upgrading to JDK8 patch 171.
[
https://issues.apache.org/jira/browse/HADOOP-15445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462146#comment-16462146
]
Ewan Higgs commented on HADOOP-15445:
-
OpenJDK will get the same feature:
https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2018-2794
> TestCryptoAdminCLI test failure when upgrading to JDK8 patch 171.
> -
>
> Key: HADOOP-15445
> URL: https://issues.apache.org/jira/browse/HADOOP-15445
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ewan Higgs
>Priority: Major
>
> JDK8 patch 171 introduces a new feature:
> {quote}
> h3. New Features
> security-libs/javax.crypto*[!https://www.oracle.com/webfolder/s/dm/st/images/lp-external-link-arrow.png!|http://www.oracle.com/technetwork/java/javase/8u171-relnotes-430.html#JDK-8189997]
> Enhanced KeyStore Mechanisms*
> A new security property named {{jceks.key.serialFilter}} has been introduced.
> If this filter is configured, the JCEKS KeyStore uses it during the
> deserialization of the encrypted Key object stored inside a SecretKeyEntry.
> If it is not configured or if the filter result is UNDECIDED (for example,
> none of the patterns match), then the filter configured by
> {{jdk.serialFilter}} is consulted.
> If the system property {{jceks.key.serialFilter}} is also supplied, it
> supersedes the security property value defined here.
> The filter pattern uses the same format as {{jdk.serialFilter}}. The default
> pattern allows {{java.lang.Enum}}, {{java.security.KeyRep}},
> {{java.security.KeyRep$Type}}, and {{javax.crypto.spec.SecretKeySpec}} but
> rejects all the others.
> Customers storing a SecretKey that does not serialize to the above types must
> modify the filter to make the key extractable.
> {quote}
> We believe this causes some test failures:
>
> {quote}{{{color:#33}java.io.IOException: Can't recover key for myKey from
> keystore
> file:/{color}{color:#33}home/{color}{color:#33}jenkins/{color}{color:#33}workspace/{color}{color:#33}hadoopFullBuild/{color}{color:#33}hadoop-hdfs-project/{color}{color:#33}hadoop-hdfs/{color}{color:#33}target/{color}{color:#33}test/{color}{color:#33}data/{color}{color:#33}53406117-0132-401e-a67d-6672f1b6a14a/{color}{color:#33}test.jks
> at
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider.getMetadata(JavaKeyStoreProvider.java:424)
> at
> org.apache.hadoop.crypto.key.KeyProviderExtension.getMetadata(KeyProviderExtension.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.ensureKeyIsInitialized(FSDirEncryptionZoneOp.java:124)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createEncryptionZone(FSNamesystem.java:7227)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createEncryptionZone(NameNodeRpcServer.java:2082)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createEncryptionZone(ClientNamenodeProtocolServerSideTranslatorPB.java:1524)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) Caused by:
> java.security.UnrecoverableKeyException: Rejected by the
> jceks.key.serialFilter or jdk.serialFilter property at
> com.sun.crypto.provider.KeyProtector.unseal(KeyProtector.java:352) at
> com.sun.crypto.provider.JceKeyStore.engineGetKey(JceKeyStore.java:136) at
> java.security.KeyStore.getKey(KeyStore.java:1023){color}}}
> {quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15445) TestCryptoAdminCLI test failure when upgrading to JDK8 patch 171.
Ewan Higgs created HADOOP-15445:
---
Summary: TestCryptoAdminCLI test failure when upgrading to JDK8
patch 171.
Key: HADOOP-15445
URL: https://issues.apache.org/jira/browse/HADOOP-15445
Project: Hadoop Common
Issue Type: Bug
Reporter: Ewan Higgs
JDK8 patch 171 introduces a new feature:
{quote}
h3. New Features
security-libs/javax.crypto*[!https://www.oracle.com/webfolder/s/dm/st/images/lp-external-link-arrow.png!|http://www.oracle.com/technetwork/java/javase/8u171-relnotes-430.html#JDK-8189997]
Enhanced KeyStore Mechanisms*
A new security property named {{jceks.key.serialFilter}} has been introduced.
If this filter is configured, the JCEKS KeyStore uses it during the
deserialization of the encrypted Key object stored inside a SecretKeyEntry. If
it is not configured or if the filter result is UNDECIDED (for example, none of
the patterns match), then the filter configured by {{jdk.serialFilter}} is
consulted.
If the system property {{jceks.key.serialFilter}} is also supplied, it
supersedes the security property value defined here.
The filter pattern uses the same format as {{jdk.serialFilter}}. The default
pattern allows {{java.lang.Enum}}, {{java.security.KeyRep}},
{{java.security.KeyRep$Type}}, and {{javax.crypto.spec.SecretKeySpec}} but
rejects all the others.
Customers storing a SecretKey that does not serialize to the above types must
modify the filter to make the key extractable.
{quote}
We believe this causes some test failures:
{quote}{{{color:#33}java.io.IOException: Can't recover key for myKey from
keystore
file:/{color}{color:#33}home/{color}{color:#33}jenkins/{color}{color:#33}workspace/{color}{color:#33}hadoopFullBuild/{color}{color:#33}hadoop-hdfs-project/{color}{color:#33}hadoop-hdfs/{color}{color:#33}target/{color}{color:#33}test/{color}{color:#33}data/{color}{color:#33}53406117-0132-401e-a67d-6672f1b6a14a/{color}{color:#33}test.jks
at
org.apache.hadoop.crypto.key.JavaKeyStoreProvider.getMetadata(JavaKeyStoreProvider.java:424)
at
org.apache.hadoop.crypto.key.KeyProviderExtension.getMetadata(KeyProviderExtension.java:100)
at
org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.ensureKeyIsInitialized(FSDirEncryptionZoneOp.java:124)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createEncryptionZone(FSNamesystem.java:7227)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createEncryptionZone(NameNodeRpcServer.java:2082)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createEncryptionZone(ClientNamenodeProtocolServerSideTranslatorPB.java:1524)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) Caused by:
java.security.UnrecoverableKeyException: Rejected by the jceks.key.serialFilter
or jdk.serialFilter property at
com.sun.crypto.provider.KeyProtector.unseal(KeyProtector.java:352) at
com.sun.crypto.provider.JceKeyStore.engineGetKey(JceKeyStore.java:136) at
java.security.KeyStore.getKey(KeyStore.java:1023){color}}}
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14188: Status: Patch Available (was: Open) Removing references to mockito whitebox in the files [~ajisakaa] mentioned. Good catch! > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Ewan Higgs >Priority: Major > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, > HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, > HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch, > HADOOP-14188.12.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14188: Status: Open (was: Patch Available) > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Ewan Higgs >Priority: Major > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, > HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, > HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch, > HADOOP-14188.12.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14188: Attachment: HADOOP-14188.12.patch > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Ewan Higgs >Priority: Major > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, > HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, > HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch, > HADOOP-14188.12.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14188: Status: Open (was: Patch Available) > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, > HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, > HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14188: Attachment: HADOOP-14188.11.patch > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, > HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, > HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14188: Status: Patch Available (was: Open) Attaching 11 patch which rebases 10 onto current trunk HEAD (626690612cd0957316628376744a8be62f891665) > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, > HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, > HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14394) Provide Builder pattern for DistributedFileSystem.create
[
https://issues.apache.org/jira/browse/HADOOP-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452076#comment-16452076
]
Ewan Higgs commented on HADOOP-14394:
-
Not sure if this is an ok place to ask or if I should take it to the dev list,
but I would like to create a mock for {{class
FileSystemDataOutputStreamBuilder}} but it's marked final. Is there a strong
reason why this is a final class? Otherwise, I just remove the final marker.
Thanks!
> Provide Builder pattern for DistributedFileSystem.create
>
>
> Key: HADOOP-14394
> URL: https://issues.apache.org/jira/browse/HADOOP-14394
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14394.00.patch, HADOOP-14394.01.patch,
> HADOOP-14394.02.patch, HADOOP-14394.03.patch, HADOOP-14394.04.patch,
> HADOOP-14394.05.patch
>
>
> This JIRA continues to refine the {{FSOutputStreamBuilder}} interface
> introduced in HDFS-11170.
> It should also provide a spec for the Builder API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15376) Remove double semi colons on imports that make Clover fall over.
[
https://issues.apache.org/jira/browse/HADOOP-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15376:
Assignee: Ewan Higgs
Status: Patch Available (was: Open)
> Remove double semi colons on imports that make Clover fall over.
>
>
> Key: HADOOP-15376
> URL: https://issues.apache.org/jira/browse/HADOOP-15376
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Trivial
> Attachments: HADOOP-15376.01.patch
>
>
> Clover will fall over if there are double semicolons on imports.
> The error looks like:
> {code:java}
> [INFO] Clover free edition.
> [INFO] Updating existing database at
> '/Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/target/clover/clover.db'.
> [INFO] Processing files at 1.8 source level.
> [INFO]
> /Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:43:1:expecting
> EOF, found 'import'
> [INFO] Instrumentation error
> com.atlassian.clover.api.CloverException:
> /Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:43:1:expecting
> EOF, found 'import'{code}
>
> Thankfully we only have one location with this:
> {code:java}
> $ find . -name \*.java -exec grep '^import .*;;' {} +
> ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:import
> org.apache.commons.io.FileUtils;;{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15376) Remove double semi colons on imports that make Clover fall over.
[
https://issues.apache.org/jira/browse/HADOOP-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15376:
Attachment: HADOOP-15376.01.patch
> Remove double semi colons on imports that make Clover fall over.
>
>
> Key: HADOOP-15376
> URL: https://issues.apache.org/jira/browse/HADOOP-15376
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Priority: Trivial
> Attachments: HADOOP-15376.01.patch
>
>
> Clover will fall over if there are double semicolons on imports.
> The error looks like:
> {code:java}
> [INFO] Clover free edition.
> [INFO] Updating existing database at
> '/Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/target/clover/clover.db'.
> [INFO] Processing files at 1.8 source level.
> [INFO]
> /Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:43:1:expecting
> EOF, found 'import'
> [INFO] Instrumentation error
> com.atlassian.clover.api.CloverException:
> /Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:43:1:expecting
> EOF, found 'import'{code}
>
> Thankfully we only have one location with this:
> {code:java}
> $ find . -name \*.java -exec grep '^import .*;;' {} +
> ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:import
> org.apache.commons.io.FileUtils;;{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15376) Remove double semi colons on imports that make Clover fall over.
Ewan Higgs created HADOOP-15376:
---
Summary: Remove double semi colons on imports that make Clover
fall over.
Key: HADOOP-15376
URL: https://issues.apache.org/jira/browse/HADOOP-15376
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Ewan Higgs
Clover will fall over if there are double semicolons on imports.
The error looks like:
{code:java}
[INFO] Clover free edition.
[INFO] Updating existing database at
'/Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/target/clover/clover.db'.
[INFO] Processing files at 1.8 source level.
[INFO]
/Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:43:1:expecting
EOF, found 'import'
[INFO] Instrumentation error
com.atlassian.clover.api.CloverException:
/Users/ehiggs/src/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:43:1:expecting
EOF, found 'import'{code}
Thankfully we only have one location with this:
{code:java}
$ find . -name \*.java -exec grep '^import .*;;' {} +
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java:import
org.apache.commons.io.FileUtils;;{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430776#comment-16430776 ] Ewan Higgs commented on HADOOP-14663: - -05 * Somehow forgot to pull in [~aw]'s work for hadoop-anotations in 04. This is fixed now. > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Attachment: HADOOP-14663.05.patch > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430769#comment-16430769 ] Ewan Higgs commented on HADOOP-14663: - Beware, extra semicolons in import statements can cause Clover to fail to instrument: [https://jira.atlassian.com/browse/CLOV-963] > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430744#comment-16430744 ] Ewan Higgs commented on HADOOP-14663: - -04 Rebased. Clover support ends on April 11 ([https://www.atlassian.com/blog/announcements/atlassian-clover-open-source).] Can we move on this or HADOOP-15190? Thanks! > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Attachment: HADOOP-14663.04.patch > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories
[
https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400153#comment-16400153
]
Ewan Higgs commented on HADOOP-15209:
-
{code:java}
+ * We do not rely on parent entries being added immediately before children,
+ * as sorting may place "/dir12" between "/dir1" and its descendants.
+ *{code}
AFAICT, SequenceFile.Sorter will put these in the correct order (for
alphanumerics... if you have (, ), #, - etc in your filename it probably gets
wonky). This means you can do the following:
{code:java}
boolean shouldDelete(CopyListingFileStatus status) {
final Path path = status.getPath();
Preconditions.checkArgument(!path.isRoot(), "Root Dir");
final String pathStr = path.toString();
final String pathAsDir = pathStr + Path.SEPARATOR;
if (lastDir == null) {
if (status.isDirectory()) {
lastDir = pathAsDir;
}
return true;
}
if (pathStr.startsWith(lastDir)
|| pathAsDir.equals(lastDir)) {
return false;
} else {
if (status.isDirectory()) {
lastDir = pathAsDir;
}
return true;
}
}{code}
This means you no longer need a cache. If you'd like I can attach a patch with
the update that passes all the unit tests.
> DistCp to eliminate needless deletion of files under already-deleted
> directories
>
>
> Key: HADOOP-15209
> URL: https://issues.apache.org/jira/browse/HADOOP-15209
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15209-001.patch, HADOOP-15209-002.patch,
> HADOOP-15209-003.patch, HADOOP-15209-004.patch, HADOOP-15209-005.patch,
> HADOOP-15209-006.patch, HADOOP-15209-007.patch
>
>
> DistCP issues a delete(file) request even if is underneath an already deleted
> directory. This generates needless load on filesystems/object stores, and, if
> the store throttles delete, can dramatically slow down the delete operation.
> If the distcp delete operation can build a history of deleted directories,
> then it will know when it does not need to issue those deletes.
> Care is needed here to make sure that whatever structure is created does not
> overload the heap of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories
[
https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397153#comment-16397153
]
Ewan Higgs commented on HADOOP-15209:
-
I tried this out on a directory with 6995 files (a hadoop distribution binary
release), writing to an S3A compatible storage, and it appears to work. The
delete part was fairly quick and only logged deletes at the directory level,
letting the FileSystem perform the delete for everything with the directory's
prefix.
Then I cleaned out the source directory and ran distcp again - and it correctly
elided all of the deletions but the top level:
{quote}{{Deleted from target: files: 0 directories: 1; skipped deletions 6995;
deletions already missing 0; failed deletes 0}}{quote}
As an aside, I seem to be unable to find where the DistCp counters are
formatted such that BANDWITH_IN_BYTES becomes "Bandwidth in Btyes":
{quote}{{ DistCp Counters
}}
{{ Bandwidth in Btyes=189349 }}
{{ Bytes Copied=312048557 }}
{{ Bytes Expected=312048557 }}
{{ Files Copied=6155 }}
{{ DIR_COPY=841 }}{quote}
> DistCp to eliminate needless deletion of files under already-deleted
> directories
>
>
> Key: HADOOP-15209
> URL: https://issues.apache.org/jira/browse/HADOOP-15209
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15209-001.patch, HADOOP-15209-002.patch,
> HADOOP-15209-003.patch, HADOOP-15209-004.patch, HADOOP-15209-005.patch,
> HADOOP-15209-006.patch, HADOOP-15209-007.patch
>
>
> DistCP issues a delete(file) request even if is underneath an already deleted
> directory. This generates needless load on filesystems/object stores, and, if
> the store throttles delete, can dramatically slow down the delete operation.
> If the distcp delete operation can build a history of deleted directories,
> then it will know when it does not need to issue those deletes.
> Care is needed here to make sure that whatever structure is created does not
> overload the heap of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395077#comment-16395077
]
Ewan Higgs commented on HADOOP-14707:
-
Reviewed. The approach looks sound given the legacy nature of the Filesystem
interface (i.e. we're not about to start breaking up Filesystem into various
interfaces/facades for the various capabilities). But Jenkins has a lot of
findbugs issues and test failures that need to be addressed.
{quote}patch is a bit over complex for the 3.1 deadline, pushing back
{quote}
Now sure how you would reduce the complexity. Did you have ideas?
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch,
> HADOOP-14707-003.patch
>
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392948#comment-16392948
]
Ewan Higgs commented on HADOOP-14707:
-
I take it we use String instead of an enum of capabilities because we don't
want to the capabilities to be part of the protobuf layer - meaning a costly
rollout process?
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch,
> HADOOP-14707-003.patch
>
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378863#comment-16378863
]
Ewan Higgs commented on HADOOP-15269:
-
For folks reading along, the netty issue is HADOOP-15264.
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365428#comment-16365428
]
Ewan Higgs edited comment on HADOOP-14943 at 2/15/18 11:40 AM:
---
[~ste...@apache.org],
{quote}You don't want location affinity in object stores, not really ... though
[~ehiggs] and [~Thomas Demoor] might have different data
{quote}
If I understand you correctly, no you don't want location affinity in object
stores.
Edit: new Jira setup doesn't like my quote tags.
was (Author: ehiggs):
[~ste...@apache.org],
{quote}You don't want location affinity in object stores, not really ... though
[~ehiggs] and [~Thomas Demoor] might have different data\{quote}
If I understand you correctly, no you don't want location affinity in object
stores.
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365428#comment-16365428
]
Ewan Higgs commented on HADOOP-14943:
-
[~ste...@apache.org],
{quote}You don't want location affinity in object stores, not really ... though
[~ehiggs] and [~Thomas Demoor] might have different data\{quote}
If I understand you correctly, no you don't want location affinity in object
stores.
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334245#comment-16334245 ] Ewan Higgs edited comment on HADOOP-13363 at 1/22/18 1:21 PM: -- For what it's worth, source builds of protobuf 2.5.0 can now fail as the automake script tries to download the googletest dependency from a stale URL. A workaround is to have googletest installed. https://github.com/google/protobuf/issues/2025 was (Author: ehiggs): For what it's worth, source builds of protobuf 2.5.0 can now fail as the automake script tries to download the googletest dependency from a stale URL. A workaround is to have googletest installed. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Tsuyoshi Ozawa >Priority: Major > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334245#comment-16334245 ] Ewan Higgs commented on HADOOP-13363: - For what it's worth, source builds of protobuf 2.5.0 can now fail as the automake script tries to download the googletest dependency from a stale URL. A workaround is to have googletest installed. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Tsuyoshi Ozawa >Priority: Major > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13625) Document FileSystem actions that trigger update of modification time.
[
https://issues.apache.org/jira/browse/HADOOP-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283354#comment-16283354
]
Ewan Higgs commented on HADOOP-13625:
-
{quote}{code}
FileSystem#createSnapshot
{code}{quote}
Obviously on the {{.snapshot/...}} directory the mtime should be the time of
creation, but should the mtime of the directory being snapshotted be updated?
This means two consecutive snapshots will not result in a 'no differences' diff
due to the differring mtimes.
> Document FileSystem actions that trigger update of modification time.
> -
>
> Key: HADOOP-13625
> URL: https://issues.apache.org/jira/browse/HADOOP-13625
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>
> Hadoop users and developers of Hadoop-compatible file systems have sometimes
> asked questions about which file system actions trigger an update of the
> path's modification time. This issue proposes to document which actions do
> and do not update modification time, so that the information is easy to find
> without reading HDFS code or manually testing individual cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14766) Cloudup: an object store high performance dfs put command
[
https://issues.apache.org/jira/browse/HADOOP-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241824#comment-16241824
]
Ewan Higgs commented on HADOOP-14766:
-
Hi Steve,
I haven't tested this, but it seems fairly straightforward based on your
description.
Is it possible to use {{java.time.Duration}} instead of the one largely copy
pasted from {{org.apache.hadoop.fs.swift.util.Duration}}?[0].
[0] This seems to be a common issue since {{org.apache.hadoop.yarn.util.Times}}
also suffers from this.
> Cloudup: an object store high performance dfs put command
> -
>
> Key: HADOOP-14766
> URL: https://issues.apache.org/jira/browse/HADOOP-14766
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14766-001.patch
>
>
> {{hdfs put local s3a://path}} is suboptimal as it treewalks down down the
> source tree then, sequentially, copies up the file through copying the file
> (opened as a stream) contents to a buffer, writes that to the dest file,
> repeats.
> For S3A that hurts because
> * it;s doing the upload inefficiently: the file can be uploaded just by
> handling the pathname to the AWS xter manager
> * it is doing it sequentially, when some parallelised upload would work.
> * as the ordering of the files to upload is a recursive treewalk, it doesn't
> spread the upload across multiple shards.
> Better:
> * build the list of files to upload
> * upload in parallel, picking entries from the list at random and spreading
> across a pool of uploaders
> * upload straight from local file (copyFromLocalFile()
> * track IO load (files created/second) to estimate risk of throttling.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206010#comment-16206010
]
Ewan Higgs commented on HADOOP-13786:
-
{quote}I should add that AFAIK Ewan has been testing the "magic" committer, not
the staging ones; his store is naturally consistent.{quote}
Yes.
{quote}What I'd like to suggest here is we create a branch for the S3Guard
phase II work (HADOOP-14825), make this the first commit & then work on the
s3guard II improvements above it. {quote}
+1
> Add S3Guard committer for zero-rename commits to S3 endpoints
> -
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch,
> HADOOP-13786-038.patch, HADOOP-13786-039.patch,
> HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch,
> HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch,
> HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch,
> HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch,
> HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch,
> HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch,
> HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch,
> HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch,
> HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch,
> HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch,
> HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch,
> HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch,
> HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch,
> HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch,
> HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch,
> HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch,
> cloud-intergration-test-failure.log, objectstore.pdf, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205884#comment-16205884
]
Ewan Higgs commented on HADOOP-13786:
-
I've been testing this mostly from a performance point of view using Hadoop MR2
using {{NullMetadataStore}} and I'm pretty happy with the results. It's indeed
twice as fast as the old style {{FileOutputCommitter}} on the system I used.
There's a lot of code here and it's been moving quite quickly but it's in
overall good shape, imo. As I'm using a {{NullMetadataStore}} a lot of the
possible error scenarios won't popup for me so it will be great if people can
cover those areas.
> Add S3Guard committer for zero-rename commits to S3 endpoints
> -
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch,
> HADOOP-13786-038.patch, HADOOP-13786-039.patch,
> HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch,
> HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch,
> HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch,
> HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch,
> HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch,
> HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch,
> HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch,
> HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch,
> HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch,
> HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch,
> HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch,
> HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch,
> HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch,
> HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch,
> HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch,
> HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch,
> cloud-intergration-test-failure.log, objectstore.pdf, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200141#comment-16200141
]
Ewan Higgs commented on HADOOP-13514:
-
[~ajisakaa], thanks for picking this up.
> Upgrade maven surefire plugin to 2.19.1
> ---
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: Ewan Higgs
>Assignee: Akira Ajisaka
> Fix For: 3.0.0
>
> Attachments: HADOOP-13514-addendum.01.patch,
> HADOOP-13514-testing.001.patch, HADOOP-13514-testing.002.patch,
> HADOOP-13514-testing.003.patch, HADOOP-13514-testing.004.patch,
> HADOOP-13514.002.patch, HADOOP-13514.003.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198513#comment-16198513
]
Ewan Higgs commented on HADOOP-13514:
-
Looks good to me.
> Upgrade maven surefire plugin to 2.19.1
> ---
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: Ewan Higgs
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13514-addendum.01.patch,
> HADOOP-13514-testing.001.patch, HADOOP-13514-testing.002.patch,
> HADOOP-13514-testing.003.patch, HADOOP-13514-testing.004.patch,
> HADOOP-13514.002.patch, HADOOP-13514.003.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-13786: Attachment: cloud-intergration-test-failure.log Log for some NullPointerExceptions after "Start iterating the provided status". > Add S3Guard committer for zero-rename commits to S3 endpoints > - > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: cloud-intergration-test-failure.log, > HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch, > HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch, > HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch, > HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch, > HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch, > HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch, > HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch, > HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch, > HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch, > HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch, > HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch, > HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch, > HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch, > HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch, > HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch, > HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch, > objectstore.pdf, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125872#comment-16125872
]
Ewan Higgs commented on HADOOP-13786:
-
Hi, I have been testing this locally with the [Hortonworks cloud-integration
project|https://github.com/hortonworks-spark/cloud-integration] and an S3
compatible backend that is has strong consistency. Because it has strong
consistency one would expect the {{NullMetadataStore}} to work. However, I'm
getting some errors.
To reproduce, I build Hadoop as follows:
{code}
mvn install -DskipShade -Dmaven.javadoc.skip=true -Pdist,parallel-tests
-DtestsThreadCount=8 -Djava.awt.headless=true -Ddeclared.hadoop.version=2.11
-DskipTests
{code}
I ran into some NPEs:
{code}
S3AFileStatus{path=s3a://s3guard-test/cloud-integration/DELAY_LISTING_ME/S3ACommitDataframeSuite/dataframe-committer/committer-default-orc/orc/part-0-8b8b323b-c747-4d72-b331-b6de1c1f8387-c000.snappy.orc;
isDirectory=false; length=2995; replication=1; blocksize=1048576;
modification_time=1502715661000; access_time=0; owner=ehiggs; group=ehiggs;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2017-08-14 15:01:02,297 [ScalaTest-main-running-S3ACommitDataframeSuite] DEBUG
s3a.S3AFileSystem (Listing.java:sourceHasNext(374)) - Start iterating the
provided status.
2017-08-14 15:01:02,304 [ScalaTest-main-running-S3ACommitDataframeSuite] ERROR
commit.S3ACommitDataframeSuite (Logging.scala:logError(91)) - After 237,747,296
nS: java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.fs.LocatedFileStatus.(LocatedFileStatus.java:87)
at
org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$listLeafFiles$3.apply(InMemoryFileIndex.scala:299)
at
org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$listLeafFiles$3.apply(InMemoryFileIndex.scala:281)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at
org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$listLeafFiles(InMemoryFileIndex.scala:281)
at
org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$1.apply(InMemoryFileIndex.scala:172)
at
org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$1.apply(InMemoryFileIndex.scala:171)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
{code}
I'll attach the trace.
> Add S3Guard committer for zero-rename commits to S3 endpoints
> -
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch,
> HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch,
> HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch,
> HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch,
> HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch,
> HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch,
> HADOOP-13786-HADOOP-13345-021.patch, HADOOP-13786-HADOOP-13345-022.patch,
> HADOOP-13786-HADOOP-13345-023.patch, HADOOP-13786-HADOOP-13345-024.patch,
> HADOOP-13786-HADOOP-13345-025.patch, HADOOP-13786-HADOOP-13345-026.patch,
> HADOOP-13786-HADOOP-13345-027.patch, HADOOP-13786-HADOOP-13345-028.patch,
> HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-029.patch,
>
