[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939015#comment-15939015
]
Steve Loughran commented on HADOOP-13786:
-----------------------------------------
And for the very curious, here's a trace of a test run of the latest code doing
some simple {{sparkContact.makeRDD(1 to
count).saveAsTextFile("s3a:///something")}} with the directory committer.
There's a job committer and task committer in the logs, hence the role log info
to work out whats happening. fs.s3a mostly at debug, except for
instrumentation. FS stats exclude all the info on the uploads because they
(currently) go straight through the s3a client.
{code}
-------------------------------------------
2017-03-23 18:42:41,455 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api)
2017-03-23 18:42:42,024 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:42,024 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:delete(1418)) - Couldn't delete
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
- does not exist
2017-03-23 18:42:42,024 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
s3.S3ANumbersSuiteV2APISuite (CloudLogging.scala:logInfo(56)) - Switching to
local file:// fs for default FS
2017-03-23 18:42:42,098 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - Running Spark version
2.2.0-SNAPSHOT
2017-03-23 18:42:42,133 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - Submitted application: test
2017-03-23 18:42:42,147 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing view acls to:
stevel
2017-03-23 18:42:42,147 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing modify acls to:
stevel
2017-03-23 18:42:42,148 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing view acls groups
to:
2017-03-23 18:42:42,149 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - Changing modify acls
groups to:
2017-03-23 18:42:42,150 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SecurityManager (Logging.scala:logInfo(54)) - SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(stevel); groups with view permissions: Set(); users with modify
permissions: Set(stevel); groups with modify permissions: Set()
2017-03-23 18:42:42,369 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
util.Utils (Logging.scala:logInfo(54)) - Successfully started service
'sparkDriver' on port 54392.
2017-03-23 18:42:42,389 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkEnv (Logging.scala:logInfo(54)) - Registering MapOutputTracker
2017-03-23 18:42:42,402 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkEnv (Logging.scala:logInfo(54)) - Registering BlockManagerMaster
2017-03-23 18:42:42,405 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) - Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2017-03-23 18:42:42,405 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) -
BlockManagerMasterEndpoint up
2017-03-23 18:42:42,412 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.DiskBlockManager (Logging.scala:logInfo(54)) - Created local directory
at
/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/blockmgr-9ab575c8-964f-4cb7-96cb-e5c6efc04d60
2017-03-23 18:42:42,429 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
memory.MemoryStore (Logging.scala:logInfo(54)) - MemoryStore started with
capacity 2004.6 MB
2017-03-23 18:42:42,466 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkEnv (Logging.scala:logInfo(54)) - Registering
OutputCommitCoordinator
2017-03-23 18:42:42,579 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
util.Utils (Logging.scala:logInfo(54)) - Successfully started service
'SparkUI' on port 4040.
2017-03-23 18:42:42,606 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
ui.SparkUI (Logging.scala:logInfo(54)) - Bound SparkUI to 0.0.0.0, and started
at http://192.168.1.38:4040
2017-03-23 18:42:42,679 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
executor.Executor (Logging.scala:logInfo(54)) - Starting executor ID driver on
host localhost
2017-03-23 18:42:42,694 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
util.Utils (Logging.scala:logInfo(54)) - Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 54394.
2017-03-23 18:42:42,695 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
netty.NettyBlockTransferService (Logging.scala:logInfo(54)) - Server created
on 192.168.1.38:54394
2017-03-23 18:42:42,696 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManager (Logging.scala:logInfo(54)) - Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
2017-03-23 18:42:42,697 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMaster (Logging.scala:logInfo(54)) - Registering
BlockManager BlockManagerId(driver, 192.168.1.38, 54394, None)
2017-03-23 18:42:42,701 [dispatcher-event-loop-2] INFO
storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) - Registering
block manager 192.168.1.38:54394 with 2004.6 MB RAM, BlockManagerId(driver,
192.168.1.38, 54394, None)
2017-03-23 18:42:42,703 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMaster (Logging.scala:logInfo(54)) - Registered
BlockManager BlockManagerId(driver, 192.168.1.38, 54394, None)
2017-03-23 18:42:42,703 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManager (Logging.scala:logInfo(54)) - Initialized BlockManager:
BlockManagerId(driver, 192.168.1.38, 54394, None)
2017-03-23 18:42:42,898 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
s3.S3ANumbersSuiteV2APISuite (CloudLogging.scala:logInfo(56)) -
Generating output under
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:42,940 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api)
2017-03-23 18:42:43,486 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:43,492 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG output.PathOutputCommitterFactory
(PathOutputCommitterFactory.java:getOutputCommitterFactory(92)) - Using
FileOutputCommitter factory class class
org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory
2017-03-23 18:42:43,495 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG commit.AbstractS3GuardCommitter
(AbstractS3GuardCommitter.java:<init>(85)) - Task committer
attempt_20170323184243_0000_m_000000_0 instantiated for job "" ID
job_20170323184243_0000 with destination
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:43,497 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG commit.AbstractS3GuardCommitter
(AbstractS3GuardCommitter.java:<init>(115)) - Task committer
attempt_20170323184243_0000_m_000000_0} instantiated for"" ID
job_20170323184243_0000
2017-03-23 18:42:43,499 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG commit.AbstractS3GuardCommitter
(AbstractS3GuardCommitter.java:setWorkPath(162)) - Setting work path to
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184243_0000_m_000000_0/_temporary/0/_temporary/attempt_20170323184243_0000_m_000000_0
2017-03-23 18:42:43,501 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
output.FileOutputCommitter (FileOutputCommitter.java:<init>(123)) - File
Output Committer Algorithm version is 1
2017-03-23 18:42:43,501 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
output.FileOutputCommitter (FileOutputCommitter.java:<init>(138)) -
FileOutputCommitter skip cleanup _temporary folders under output
directory:false, ignore cleanup failures: true
2017-03-23 18:42:43,502 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:postCreationActions(168)) - Conflict resolution
mode: FAIL
2017-03-23 18:42:43,502 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
commit.Abstract3GuardCommitterFactory
(Abstract3GuardCommitterFactory.java:createOutputCommitter(61)) - Using
Commmitter StagingS3GuardCommitter{AbstractS3GuardCommitter{role=Task committer
attempt_20170323184243_0000_m_000000_0outputPath=s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api,
workPath=file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184243_0000_m_000000_0/_temporary/0/_temporary/attempt_20170323184243_0000_m_000000_0},
finalOutputPath=null, conflictResolution=FAIL,
wrappedCommitter=FileOutputCommitter{outputPath=file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/stevel/job_20170323184243_0000/staging-uploads,
workPath=null, algorithmVersion=1, skipCleanup=false,
ignoreCleanupFailures=true}} for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:43,502 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:setupJob(520)) - Task committer
attempt_20170323184243_0000_m_000000_0, Setting up job job_20170323184242_0001
2017-03-23 18:42:43,539 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:getOutputPath(1057)) - Task committer
attempt_20170323184243_0000_m_000000_0: final output path is
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:43,539 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api)
2017-03-23 18:42:44,084 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:44,122 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - Starting job: runJob at
SparkHadoopMapReduceWriter.scala:88
2017-03-23 18:42:44,131 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - Got job 0 (runJob at
SparkHadoopMapReduceWriter.scala:88) with 1 output partitions
2017-03-23 18:42:44,132 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - Final stage: ResultStage 0 (runJob at
SparkHadoopMapReduceWriter.scala:88)
2017-03-23 18:42:44,132 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - Parents of final stage: List()
2017-03-23 18:42:44,133 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - Missing parents: List()
2017-03-23 18:42:44,137 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - Submitting ResultStage 0 (MapPartitionsRDD[1] at
map at NumbersRddTests.scala:120), which has no missing parents
2017-03-23 18:42:44,208 [dag-scheduler-event-loop] INFO memory.MemoryStore
(Logging.scala:logInfo(54)) - Block broadcast_0 stored as values in memory
(estimated size 74.9 KB, free 2004.5 MB)
2017-03-23 18:42:44,232 [dag-scheduler-event-loop] INFO memory.MemoryStore
(Logging.scala:logInfo(54)) - Block broadcast_0_piece0 stored as bytes in
memory (estimated size 25.0 KB, free 2004.5 MB)
2017-03-23 18:42:44,234 [dispatcher-event-loop-4] INFO
storage.BlockManagerInfo (Logging.scala:logInfo(54)) - Added broadcast_0_piece0
in memory on 192.168.1.38:54394 (size: 25.0 KB, free: 2004.6 MB)
2017-03-23 18:42:44,236 [dag-scheduler-event-loop] INFO spark.SparkContext
(Logging.scala:logInfo(54)) - Created broadcast 0 from broadcast at
DAGScheduler.scala:1002
2017-03-23 18:42:44,251 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - Submitting 1 missing tasks from ResultStage 0
(MapPartitionsRDD[1] at map at NumbersRddTests.scala:120) (first 15 tasks are
for partitions Vector(0))
2017-03-23 18:42:44,252 [dag-scheduler-event-loop] INFO
scheduler.TaskSchedulerImpl (Logging.scala:logInfo(54)) - Adding task set 0.0
with 1 tasks
2017-03-23 18:42:44,281 [dispatcher-event-loop-5] INFO
scheduler.TaskSetManager (Logging.scala:logInfo(54)) - Starting task 0.0 in
stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7598
bytes)
2017-03-23 18:42:44,286 [Executor task launch worker for task 0] INFO
executor.Executor (Logging.scala:logInfo(54)) - Running task 0.0 in stage 0.0
(TID 0)
2017-03-23 18:42:44,333 [Executor task launch worker for task 0] DEBUG
output.PathOutputCommitterFactory
(PathOutputCommitterFactory.java:getOutputCommitterFactory(92)) - Using
FileOutputCommitter factory class class
org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory
2017-03-23 18:42:44,334 [Executor task launch worker for task 0] DEBUG
commit.AbstractS3GuardCommitter (AbstractS3GuardCommitter.java:<init>(85)) -
Task committer attempt_20170323184242_0000_r_000000_0 instantiated for job ""
ID job_20170323184242_0000 with destination
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:44,335 [Executor task launch worker for task 0] DEBUG
commit.AbstractS3GuardCommitter (AbstractS3GuardCommitter.java:<init>(115)) -
Task committer attempt_20170323184242_0000_r_000000_0} instantiated for"" ID
job_20170323184242_0000
2017-03-23 18:42:44,336 [Executor task launch worker for task 0] DEBUG
commit.AbstractS3GuardCommitter
(AbstractS3GuardCommitter.java:setWorkPath(162)) - Setting work path to
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,337 [Executor task launch worker for task 0] INFO
output.FileOutputCommitter (FileOutputCommitter.java:<init>(123)) - File Output
Committer Algorithm version is 1
2017-03-23 18:42:44,337 [Executor task launch worker for task 0] INFO
output.FileOutputCommitter (FileOutputCommitter.java:<init>(138)) -
FileOutputCommitter skip cleanup _temporary folders under output
directory:false, ignore cleanup failures: true
2017-03-23 18:42:44,337 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:postCreationActions(168)) - Conflict resolution
mode: FAIL
2017-03-23 18:42:44,338 [Executor task launch worker for task 0] INFO
commit.Abstract3GuardCommitterFactory
(Abstract3GuardCommitterFactory.java:createOutputCommitter(61)) - Using
Commmitter StagingS3GuardCommitter{AbstractS3GuardCommitter{role=Task committer
attempt_20170323184242_0000_r_000000_0outputPath=s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api,
workPath=file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0},
finalOutputPath=null, conflictResolution=FAIL,
wrappedCommitter=FileOutputCommitter{outputPath=file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/stevel/job_20170323184243_0000/staging-uploads,
workPath=null, algorithmVersion=1, skipCleanup=false,
ignoreCleanupFailures=true}} for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:44,339 [Executor task launch worker for task 0] INFO
commit.DurationInfo (DurationInfo.java:<init>(33)) - Starting Task committer
attempt_20170323184242_0000_r_000000_0: task
attempt_20170323184242_0000_r_000000_0: creating task attempt path
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,386 [Executor task launch worker for task 0] INFO
commit.DurationInfo (DurationInfo.java:close(44)) - Task committer
attempt_20170323184242_0000_r_000000_0: task
attempt_20170323184242_0000_r_000000_0: creating task attempt path
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
: 0:00:047
2017-03-23 18:42:44,388 [Executor task launch worker for task 0] DEBUG
output.PathOutputCommitterFactory
(PathOutputCommitterFactory.java:getOutputCommitterFactory(92)) - Using
FileOutputCommitter factory class class
org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory
2017-03-23 18:42:44,388 [Executor task launch worker for task 0] DEBUG
commit.AbstractS3GuardCommitter (AbstractS3GuardCommitter.java:<init>(85)) -
Task committer attempt_20170323184242_0000_r_000000_0 instantiated for job ""
ID job_20170323184242_0000 with destination
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:44,388 [Executor task launch worker for task 0] DEBUG
commit.AbstractS3GuardCommitter (AbstractS3GuardCommitter.java:<init>(115)) -
Task committer attempt_20170323184242_0000_r_000000_0} instantiated for"" ID
job_20170323184242_0000
2017-03-23 18:42:44,389 [Executor task launch worker for task 0] DEBUG
commit.AbstractS3GuardCommitter
(AbstractS3GuardCommitter.java:setWorkPath(162)) - Setting work path to
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,390 [Executor task launch worker for task 0] INFO
output.FileOutputCommitter (FileOutputCommitter.java:<init>(123)) - File Output
Committer Algorithm version is 1
2017-03-23 18:42:44,390 [Executor task launch worker for task 0] INFO
output.FileOutputCommitter (FileOutputCommitter.java:<init>(138)) -
FileOutputCommitter skip cleanup _temporary folders under output
directory:false, ignore cleanup failures: true
2017-03-23 18:42:44,390 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:postCreationActions(168)) - Conflict resolution
mode: FAIL
2017-03-23 18:42:44,390 [Executor task launch worker for task 0] INFO
commit.Abstract3GuardCommitterFactory
(Abstract3GuardCommitterFactory.java:createOutputCommitter(61)) - Using
Commmitter StagingS3GuardCommitter{AbstractS3GuardCommitter{role=Task committer
attempt_20170323184242_0000_r_000000_0outputPath=s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api,
workPath=file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0},
finalOutputPath=null, conflictResolution=FAIL,
wrappedCommitter=FileOutputCommitter{outputPath=file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/stevel/job_20170323184243_0000/staging-uploads,
workPath=null, algorithmVersion=1, skipCleanup=false,
ignoreCleanupFailures=true}} for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:44,448 [Executor task launch worker for task 0] INFO
commit.DurationInfo (DurationInfo.java:<init>(33)) - Starting Task committer
attempt_20170323184242_0000_r_000000_0: needsTaskCommit() Task
attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,454 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:needsTaskCommit(849)) - 1 files to commit
2017-03-23 18:42:44,455 [Executor task launch worker for task 0] INFO
commit.DurationInfo (DurationInfo.java:close(44)) - Task committer
attempt_20170323184242_0000_r_000000_0: needsTaskCommit() Task
attempt_20170323184242_0000_r_000000_0: 0:00:006
2017-03-23 18:42:44,457 [Executor task launch worker for task 0] INFO
commit.DurationInfo (DurationInfo.java:<init>(33)) - Starting Task committer
attempt_20170323184242_0000_r_000000_0: commit task
attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,457 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:getTaskOutput(419)) - Scanning
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
for files to commit
2017-03-23 18:42:44,468 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:commitTaskInternal(885)) - Task committer
attempt_20170323184242_0000_r_000000_0: commitTaskInternal
2017-03-23 18:42:44,468 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:commitTaskInternal(890)) - Task committer
attempt_20170323184242_0000_r_000000_0: attempt path is
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,468 [Executor task launch worker for task 0] INFO
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:commitTaskInternal(903)) - Task committer
attempt_20170323184242_0000_r_000000_0: uploading from staging directory to S3
2017-03-23 18:42:44,468 [Executor task launch worker for task 0] INFO
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:commitTaskInternal(904)) - Task committer
attempt_20170323184242_0000_r_000000_0: Saving pending data information to
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/stevel/job_20170323184243_0000/staging-uploads/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0
2017-03-23 18:42:44,469 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:getOutputPath(1057)) - Task committer
attempt_20170323184242_0000_r_000000_0: final output path is
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:44,493 [Executor task launch worker for task 0] DEBUG
staging.S3Util (S3Util.java:multipartUpload(155)) - Initiating multipart upload
from
/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/test/mapred/local/job_20170323184243_0000/0/attempt_20170323184242_0000_r_000000_0/_temporary/0/_temporary/attempt_20170323184242_0000_r_000000_0/part-r-00000
to
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000
partition=null
2017-03-23 18:42:44,684 [Executor task launch worker for task 0] DEBUG
staging.S3Util (S3Util.java:multipartUpload(177)) - File size is 515303, number
of parts to upload = 1
2017-03-23 18:42:46,352 [Executor task launch worker for task 0] DEBUG
staging.StagingS3GuardCommitter (StagingS3GuardCommitter.java:run(939)) - Task
committer attempt_20170323184242_0000_r_000000_0: adding pending commit
PendingUpload{ to
's3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000'
,
uploadId='_P7n0N2FxTdVYiTBBFNB2Y9TOQgnbuKT1Lt.mwfx4pdmT6aK4E5XlxfpFZNwSUv0viLzNx45qvhJAqDyhSDqajozcfuFesRF2enq.bpkfc0-'}
2017-03-23 18:42:46,360 [Executor task launch worker for task 0] INFO
output.FileOutputCommitter (FileOutputCommitter.java:commitTask(582)) - Saved
output of task 'attempt_20170323184242_0000_r_000000_0' to
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/stevel/job_20170323184243_0000/staging-uploads/_temporary/0/task_20170323184242_0000_r_000000
2017-03-23 18:42:46,362 [Executor task launch worker for task 0] INFO
staging.StagingS3GuardCommitter (StagingS3GuardCommitter.java:commitTask(866))
- Task committer attempt_20170323184242_0000_r_000000_0: upload file count: 1
2017-03-23 18:42:46,362 [Executor task launch worker for task 0] INFO
commit.DurationInfo (DurationInfo.java:close(44)) - Task committer
attempt_20170323184242_0000_r_000000_0: commit task
attempt_20170323184242_0000_r_000000_0: 0:01:906
2017-03-23 18:42:46,363 [Executor task launch worker for task 0] INFO
mapred.SparkHadoopMapRedUtil (Logging.scala:logInfo(54)) -
attempt_20170323184242_0000_r_000000_0: Committed
2017-03-23 18:42:46,374 [Executor task launch worker for task 0] INFO
executor.Executor (Logging.scala:logInfo(54)) - Finished task 0.0 in stage 0.0
(TID 0). 1306 bytes result sent to driver
2017-03-23 18:42:46,380 [task-result-getter-0] INFO scheduler.TaskSetManager
(Logging.scala:logInfo(54)) - Finished task 0.0 in stage 0.0 (TID 0) in 2111 ms
on localhost (executor driver) (1/1)
2017-03-23 18:42:46,381 [task-result-getter-0] INFO
scheduler.TaskSchedulerImpl (Logging.scala:logInfo(54)) - Removed TaskSet 0.0,
whose tasks have all completed, from pool
2017-03-23 18:42:46,384 [dag-scheduler-event-loop] INFO scheduler.DAGScheduler
(Logging.scala:logInfo(54)) - ResultStage 0 (runJob at
SparkHadoopMapReduceWriter.scala:88) finished in 2.124 s
2017-03-23 18:42:46,388 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
scheduler.DAGScheduler (Logging.scala:logInfo(54)) - Job 0 finished: runJob at
SparkHadoopMapReduceWriter.scala:88, took 2.265104 s
2017-03-23 18:42:46,389 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
commit.DurationInfo (DurationInfo.java:<init>(33)) - Starting Task committer
attempt_20170323184243_0000_m_000000_0: preparing to commit Job
2017-03-23 18:42:46,390 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:getThreadPool(1094)) - Task committer
attempt_20170323184243_0000_m_000000_0: creating thread pool of size 8
2017-03-23 18:42:46,393 [s3-committer-pool-0] DEBUG staging.S3Util
(S3Util.java:readPendingCommits(234)) - Reading pending commits in file
file:/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/stevel/job_20170323184243_0000/staging-uploads/_temporary/0/task_20170323184242_0000_r_000000
2017-03-23 18:42:46,406 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api)
2017-03-23 18:42:47,006 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
2017-03-23 18:42:47,006 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
commit.DurationInfo (DurationInfo.java:close(44)) - Task committer
attempt_20170323184243_0000_m_000000_0: preparing to commit Job: 0:00:617
2017-03-23 18:42:47,007 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
commit.DurationInfo (DurationInfo.java:<init>(33)) - Starting Task committer
attempt_20170323184243_0000_m_000000_0: committing Job job_20170323184242_0001
2017-03-23 18:42:47,007 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG staging.StagingS3GuardCommitter
(StagingS3GuardCommitter.java:commitJobInternal(649)) - Task committer
attempt_20170323184243_0000_m_000000_0: committing the output of 1 task(s)
2017-03-23 18:42:47,010 [s3-committer-pool-1] DEBUG staging.S3Util
(S3Util.java:finishCommit(101)) - Finish PendingUpload{ to
's3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000'
,
uploadId='_P7n0N2FxTdVYiTBBFNB2Y9TOQgnbuKT1Lt.mwfx4pdmT6aK4E5XlxfpFZNwSUv0viLzNx45qvhJAqDyhSDqajozcfuFesRF2enq.bpkfc0-'}
2017-03-23 18:42:47,364 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary)
2017-03-23 18:42:47,941 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary
2017-03-23 18:42:47,942 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:delete(1418)) - Couldn't delete
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary
- does not exist
2017-03-23 18:42:47,943 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
commit.DurationInfo (DurationInfo.java:close(44)) - Task committer
attempt_20170323184243_0000_m_000000_0: committing Job job_20170323184242_0001:
0:00:936
2017-03-23 18:42:47,943 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG commit.FileCommitActions (FileCommitActions.java:touchSuccessMarker(231))
- Touching success marker for job
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS
2017-03-23 18:42:47,944 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS)
2017-03-23 18:42:48,494 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS
2017-03-23 18:42:48,495 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG commit.DelayedCommitFSIntegration
(DelayedCommitFSIntegration.java:getTracker(98)) - Created
DefaultPutTracker{destKey='spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS'}
2017-03-23 18:42:48,498 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:createWriteOperationHelper(703)) -
Created WriteOperationHelper {bucket=hwdev-steve-new,
key='spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS'}
2017-03-23 18:42:48,514 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:startUpload(324)) - Start
datablock[1] upload
2017-03-23 18:42:48,514 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3ADataBlocks (S3ADataBlocks.java:enterState(231)) -
FileBlock{index=1,
destFile=/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/s3ablock-0001-4746647579488981641.tmp,
state=Writing, dataSize=0, limit=8388608}: entering state Upload
2017-03-23 18:42:48,518 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:putObjectDirect(1280)) - PUT 0 bytes to
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS
2017-03-23 18:42:48,519 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:incrementPutStartStatistics(1321)) - PUT start 0 bytes
2017-03-23 18:42:48,740 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:incrementPutCompletedStatistics(1338)) - PUT completed
success=true; 0 bytes
2017-03-23 18:42:48,740 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:finishedWrite(2183)) - Finished write to
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS, len 0
2017-03-23 18:42:48,938 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks
(S3ADataBlocks.java:enterState(231)) - FileBlock{index=1,
destFile=/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/s3ablock-0001-4746647579488981641.tmp,
state=Upload, dataSize=0, limit=8388608}: entering state Closed
2017-03-23 18:42:48,938 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks
(S3ADataBlocks.java:close(346)) - Closed FileBlock{index=1,
destFile=/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/s3ablock-0001-4746647579488981641.tmp,
state=Closed, dataSize=0, limit=8388608}
2017-03-23 18:42:48,938 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks
(S3ADataBlocks.java:innerClose(886)) - Closing FileBlock{index=1,
destFile=/Users/stevel/Projects/sparkwork/spark-cloud-examples/cloud-examples/target/tmp/s3ablock-0001-4746647579488981641.tmp,
state=Closed, dataSize=0, limit=8388608}
2017-03-23 18:42:48,939 [s3a-transfer-shared-pool1-t1] DEBUG s3a.S3ADataBlocks
(S3ADataBlocks.java:closeBlock(940)) - block[1]: closeBlock()
2017-03-23 18:42:48,939 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:writeSuccessful(2701)) - Successful
write to
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS, len 0
2017-03-23 18:42:48,941 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary-1
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary-1)
2017-03-23 18:42:50,072 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1905)) - Not Found:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary-1
2017-03-23 18:42:50,072 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:delete(1418)) - Couldn't delete
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_temporary-1
- does not exist
2017-03-23 18:42:50,074 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
io.SparkHadoopMapReduceWriter (Logging.scala:logInfo(54)) - Job
job_20170323184242_0001 committed.
2017-03-23 18:42:50,081 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
ui.SparkUI (Logging.scala:logInfo(54)) - Stopped Spark web UI at
http://192.168.1.38:4040
2017-03-23 18:42:50,108 [dispatcher-event-loop-2] INFO
spark.MapOutputTrackerMasterEndpoint (Logging.scala:logInfo(54)) -
MapOutputTrackerMasterEndpoint stopped!
2017-03-23 18:42:50,117 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
memory.MemoryStore (Logging.scala:logInfo(54)) - MemoryStore cleared
2017-03-23 18:42:50,118 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManager (Logging.scala:logInfo(54)) - BlockManager stopped
2017-03-23 18:42:50,123 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
storage.BlockManagerMaster (Logging.scala:logInfo(54)) - BlockManagerMaster
stopped
2017-03-23 18:42:50,125 [dispatcher-event-loop-0] INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint
(Logging.scala:logInfo(54)) - OutputCommitCoordinator stopped!
2017-03-23 18:42:50,128 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - Successfully stopped
SparkContext
2017-03-23 18:42:50,130 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
s3.S3ANumbersSuiteV2APISuite (CloudLogging.scala:logInfo(56)) - Filesystem
statistics
S3AFileSystem
{uri=s3a://hwdev-steve-new, workingDir=s3a://hwdev-steve-new/user/stevel,
inputPolicy=sequential, partSize=8388608, enableMultiObjectsDelete=true,
maxKeys=5000, readAhead=65536, blockSize=1048576,
multiPartThreshold=2147483647, serverSideEncryptionAlgorithm='NONE',
blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@6707a4bf,
metastore=NullMetadataStore, authoritative=false,
boundedExecutor=BlockingThreadPoolExecutorService
{SemaphoredDelegatingExecutor
{permitCount=25, available=25, waiting=0}, activeCount=0},
unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@3fc7c734[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0],
statistics
{0 bytes read, 0 bytes written, 7 read ops, 0 large read ops, 3 write ops},
metrics
{
{Context=S3AFileSystem}
{FileSystemId=58e091c4-0ae9-4626-8ecf-0d1ac3ffdb27-hwdev-steve-new}
{fsURI=s3a://hwdev-steve-new}
{files_created=1}
{files_copied=0}
{files_copied_bytes=0}
{files_deleted=0}
{fake_directories_deleted=3}
{directories_created=0}
{directories_deleted=0}
{ignored_errors=3}
{op_copy_from_local_file=0}
{op_exists=3}
{op_get_file_status=7}
{op_glob_status=0}
{op_is_directory=0}
{op_is_file=0}
{op_list_files=0}
{op_list_located_status=0}
{op_list_status=0}
{op_mkdirs=0}
{op_rename=0}
{object_copy_requests=0}
{object_delete_requests=1}
{object_list_requests=7}
{object_continue_list_requests=0}
{object_metadata_requests=14}
{object_multipart_aborted=0}
{object_put_bytes=0}
{object_put_requests=1}
{object_put_requests_completed=1}
{stream_write_failures=0}
{stream_write_block_uploads=0}
{stream_write_block_uploads_committed=0}
{stream_write_block_uploads_aborted=0}
{stream_write_total_time=0}
{stream_write_total_data=0}
{object_put_requests_active=0}
{object_put_bytes_pending=0}
{stream_write_block_uploads_active=0}
{stream_write_block_uploads_pending=1}
{stream_write_block_uploads_data_pending=0}
{stream_read_fully_operations=0}
{stream_opened=0}
{stream_bytes_skipped_on_seek=0}
{stream_closed=0}
{stream_bytes_backwards_on_seek=0}
{stream_bytes_read=0}
{stream_read_operations_incomplete=0}
{stream_bytes_discarded_in_abort=0}
{stream_close_operations=0}
{stream_read_operations=0}
{stream_aborted=0}
{stream_forward_seek_operations=0}
{stream_backward_seek_operations=0}
{stream_seek_operations=0}
{stream_bytes_read_in_close=0}
{stream_read_exceptions=0} }}
2017-03-23 18:42:50,130 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:listFiles(2505)) -
listFiles(s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api,
true)
2017-03-23 18:42:50,130 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api
(spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api)
2017-03-23 18:42:50,761 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1880)) - Found path
as directory (with /): 0/1
2017-03-23 18:42:50,761 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1885)) - Summary:
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS 0
2017-03-23 18:42:50,763 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:listFiles(2518)) - Requesting all
entries under spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/
with delimiter 'null'
2017-03-23 18:42:50,953 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (Listing.java:buildNextStatusBatch(296)) -
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS:
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS size=0
2017-03-23 18:42:50,954 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (Listing.java:buildNextStatusBatch(302)) - Adding:
S3AFileStatus{path=s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS;
isDirectory=false; length=0; replication=1; blocksize=1048576;
modification_time=1490294569000; access_time=0; owner=stevel; group=stevel;
permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=FALSE
2017-03-23 18:42:50,954 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (Listing.java:buildNextStatusBatch(296)) -
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000:
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000
size=515303
2017-03-23 18:42:50,955 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (Listing.java:buildNextStatusBatch(302)) - Adding:
S3AFileStatus{path=s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000;
isDirectory=false; length=515303; replication=1; blocksize=1048576;
modification_time=1490294565000; access_time=0; owner=stevel; group=stevel;
permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=FALSE
2017-03-23 18:42:50,955 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (Listing.java:buildNextStatusBatch(330)) - Added 2
entries; ignored 0; hasNext=true; hasMoreObjects=false
2017-03-23 18:42:50,957 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
s3.S3ANumbersSuiteV2APISuite (CloudLogging.scala:logInfo(56)) - Contents of
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api:
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS
[0]
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000
[515303]
Dirs: 0, Files: 2, total bytes = 515303
2017-03-23 18:42:50,964 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite] INFO
spark.SparkContext (Logging.scala:logInfo(54)) - SparkContext already stopped.
2017-03-23 18:42:50,966 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite
(spark-cloud/S3ANumbersSuiteV2APISuite)
- SaveRDD
+ Cleaning s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite
2017-03-23 18:42:51,514 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1880)) - Found path
as directory (with /): 1/0
2017-03-23 18:42:51,514 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1888)) - Prefix:
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/
2017-03-23 18:42:51,514 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1782)) - Getting
path status for s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite
(spark-cloud/S3ANumbersSuiteV2APISuite)
2017-03-23 18:42:52,059 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1880)) - Found path
as directory (with /): 1/0
2017-03-23 18:42:52,059 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1888)) - Prefix:
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/
2017-03-23 18:42:52,060 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1444)) - Delete path
s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite - recursive true
2017-03-23 18:42:52,060 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1449)) - delete: Path
is a directory: s3a://hwdev-steve-new/spark-cloud/S3ANumbersSuiteV2APISuite
2017-03-23 18:42:52,060 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1473)) - Getting
objects for directory prefix spark-cloud/S3ANumbersSuiteV2APISuite/ to delete
2017-03-23 18:42:52,248 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1483)) - Got object to
delete spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/_SUCCESS
2017-03-23 18:42:52,248 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1483)) - Got object to
delete
spark-cloud/S3ANumbersSuiteV2APISuite/numbers_rdd_tests_v2api/part-r-00000-job_20170323184243_0000
2017-03-23 18:42:53,499 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1880)) - Found path
as directory (with /): 1/0
2017-03-23 18:42:53,500 [ScalaTest-main-running-S3ANumbersSuiteV2APISuite]
DEBUG s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(1888)) - Prefix:
spark-cloud/S3ALineCountWritebackSuite/
Run completed in 15 seconds, 386 milliseconds.
{code}
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch,
> HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch,
> HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch,
> HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch,
> HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch,
> HADOOP-13786-HADOOP-13345-019.patch, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
