[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15519178#comment-15519178
]
Steve Loughran commented on HADOOP-13560:
-----------------------------------------
last patch 7129f against s3a ireland *with block output enabled on all S3a
tests by default*
Failure in distcp
{code}
-------------------------------------------------------------------------------Tests
run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 110.45 sec <<<
FAILURE! - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCplargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp)
Time elapsed: 38.58 sec <<< FAILURE! java.lang.AssertionError: wrong length
DeprecatedRawLocalFileStatus{path=file:/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test/data/ITestS3AContractDistCp/largeFilesFromRemote/outputDir/inputDir/file1;
isDirectory=false; length=1048576; replication=1; blocksize=33554432;
modification_time=1474617816000; access_time=1474617816000; owner=; group=;
permission=rw-rw-rw-; isSymlink=false} expected:<11534336> but was:<1048576>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.fs.contract.ContractTestUtils.verifyFileContents(ContractTestUtils.java:202)
at
org.apache.hadoop.tools.contract.AbstractContractDistCpTest.largeFiles(AbstractContractDistCpTest.java:174)
at
org.apache.hadoop.tools.contract.AbstractContractDistCpTest.largeFilesFromRemote(AbstractContractDistCpTest.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
target/failsafe-reports/org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp.txt
(END)
{code}
This is interesting; it is not occuring when run standalone. Race condition or
consistency problem surfacing? Or did this patch break upload when block output
is enabled everywhere. I may try to add some checks before the discp D/L to
verify the source files are of the expected files, so guarantee it's not a
consistency.
> S3ABlockOutputStream to support huge (many GB) file writes
> ----------------------------------------------------------
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13560-branch-2-001.patch,
> HADOOP-13560-branch-2-002.patch, HADOOP-13560-branch-2-003.patch,
> HADOOP-13560-branch-2-004.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 2. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
