[
https://issues.apache.org/jira/browse/HADOOP-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16560023#comment-16560023
]
Steve Loughran commented on HADOOP-15634:
-----------------------------------------
bq if it is a simple pull (from an S3 location) and write (to an S3 location),
then why do we need local disk
There's an S3 COPY command which can do in-store copy at ~6-10 MB/s. We don't
expose that in the Hadoop FS APIs, though we use this operation for mimicing
in-bucket renames
Otherwise, a copy from s3a://src to s3a://dest downloads locally and uploads;
the upload code is:
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
I belive AWS's s3 distcp does things direct, and perhaps CircusTrain from
expedia does this too. If you are trying to use the open source distcp, there
are opportunities to speedup using cloud storage as a destination, and between
cloud stores of the same cloud infra. If you want to help, you are welcome to
contribute!
> LocalDirAllocator using up local nonDFS when set to S3
> ------------------------------------------------------
>
> Key: HADOOP-15634
> URL: https://issues.apache.org/jira/browse/HADOOP-15634
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.3
> Environment: EMR-5.15, Hadoop-2.8.3, Hive-2.3.3, Tez-0.8.4, Beeline.
> Target table is defined for ACID transactions with location on S3.
> Insert source table is on S3.
> Reporter: Phani Kondapalli
> Priority: Blocker
>
> Manually modified the yarn-site.xml from within the EMR, set the param
> yarn.nodemanager.local-dirs to point to s3, reloaded the services on Master
> and Core nodes. Disk seemed to stay intact but hdfs dfsadmin -report showed
> nonDFS usage and then finally it failed with below error.
> Error: org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1532581073633_0001_2_00, diagnostics=[Task failed,
> taskId=task_1532581073633_0001_2_00_000898, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1532581073633_0001_2_00_000898_0:org.apache.hadoop.util.DiskChecker$DiskErrorException:
> Could not find any valid local directory for
> output/attempt_1532581073633_0001_2_00_000898_0_10013_1/file.out
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at
> org.apache.tez.runtime.library.common.task.local.output.TezTaskOutputFiles.getSpillFileForWrite(TezTaskOutputFiles.java:207)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.spill(PipelinedSorter.java:545)
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
