[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210344#comment-16210344
]
Aaron Fabbri commented on HADOOP-13786:
---------------------------------------
In terms of testing, one issue I'm hitting on OS X is the ITMagicCommitMRJob
test seems to hang. The output keeps complaining about du command failure
(pasted below). Looking into it. Shout if you've seen anything similar.
{noformat}
2017-10-18 15:58:58,914 [RM Event dispatcher] INFO attempt.RMAppAttemptImpl
(RMAppAttemptImpl.java:handle(918)) - appattempt_1508367534738_0001_000001
State change from NEW to SUBMITTED on event = START
2017-10-18 15:58:58,930 [RM Event dispatcher] INFO attempt.RMAppAttemptImpl
(RMAppAttemptImpl.java:handle(918)) - appattempt_1508367534738_0001_000001
State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED
2017-10-18 15:58:59,092 [JUnit-testMRJob] INFO impl.YarnClientImpl
(YarnClientImpl.java:submitApplication(295)) - Submitted application
application_1508367534738_0001
2017-10-18 15:58:59,117 [JUnit-testMRJob] INFO mapreduce.Job
(Job.java:submit(1574)) - The url to track the job:
http://fabbri-laptop.lan:0/proxy/application_1508367534738_0001/
2017-10-18 15:58:59,117 [JUnit-testMRJob] INFO commit.DurationInfo
(DurationInfo.java:<init>(44)) - Starting: Job Execution
2017-10-18 15:58:59,118 [JUnit-testMRJob] INFO mapreduce.Job
(Job.java:monitorAndPrintJob(1619)) - Running job: job_1508367534738_0001
2017-10-18 15:59:23,240 [Log Scanner/Cleaner #0] INFO hs.JobHistory
(JobHistory.java:run(201)) - History Cleaner started
2017-10-18 15:59:23,255 [Log Scanner/Cleaner #0] INFO hs.JobHistory
(JobHistory.java:run(207)) - History Cleaner complete
2017-10-18 16:01:53,232 [Log Scanner/Cleaner #1] INFO hs.JobHistory
(JobHistory.java:run(191)) - Starting scan to move intermediate done files
2017-10-18 16:08:19,766
[refreshUsed-/Users/fabbri/Code/hadoop/hadoop-tools/hadoop-aws/target/test/data/dfs/data/data1/current/BP-135722307-192.168.0.110-1508367530352]
WARN fs.CachingGetSpaceUsed (DU.java:refresh(55)) - Could not get disk usage
information for path
/Users/fabbri/Code/hadoop/hadoop-tools/hadoop-aws/target/test/data/dfs/data/data1/current/BP-135722307-192.168.0.110-1508367530352
java.io.IOException: Expecting a line not the end of stream
at org.apache.hadoop.fs.DU$DUShell.parseExecResult(DU.java:79)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:980)
at org.apache.hadoop.util.Shell.run(Shell.java:887)
at org.apache.hadoop.fs.DU$DUShell.startRefresh(DU.java:62)
at org.apache.hadoop.fs.DU.refresh(DU.java:53)
at
org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:181)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Add S3Guard committer for zero-rename commits to S3 endpoints
> -------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch,
> HADOOP-13786-038.patch, HADOOP-13786-039.patch,
> HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch,
> HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch,
> HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch,
> HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch,
> HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch,
> HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch,
> HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch,
> HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch,
> HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch,
> HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch,
> HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch,
> HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch,
> HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch,
> HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch,
> HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch,
> HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch,
> cloud-intergration-test-failure.log, objectstore.pdf, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
