[jira] [Commented] (HADOOP-13569) S3AFastOutputStream to take ProgressListener in file create()

2017-01-04 Thread Junping Du (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799399#comment-15799399
 ] 

Junping Du commented on HADOOP-13569:
-

Drop fix version for duplicated JIRAs.

> S3AFastOutputStream to take ProgressListener in file create()
> -
>
> Key: HADOOP-13569
> URL: https://issues.apache.org/jira/browse/HADOOP-13569
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> For scale testing I'd like more meaningful progress than the Hadoop 
> {{Progressable}} offers. 
> Proposed: having {{S3AFastOutputStream}} check to see if the progressable 
> passed in is also an instance of {{com.amazonaws.event.ProgressListener}} 
> —and if so, wire it up directly.
> This allows tests to directly track state of upload, log it and perhaps even 
> assert on it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13569) S3AFastOutputStream to take ProgressListener in file create()

2016-08-31 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452550#comment-15452550
 ] 

Steve Loughran commented on HADOOP-13569:
-

What the output would look like
{code}
2016-08-31 16:22:15,353 [JUnit] INFO  scale.STestS3AHugeFileCreate 
(STestS3AHugeFileCreate.java:test_010_CreateHugeFile(142)) - [100%] Written 
256.00 MB out of 256.00 MB; PUT = 209715200 bytes in 2 operations
2016-08-31 16:22:15,353 [JUnit] INFO  scale.STestS3AHugeFileCreate 
(STestS3AHugeFileCreate.java:test_010_CreateHugeFile(154)) - Closing file and 
completing write operation
2016-08-31 16:22:15,372 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event TRANSFER_PART_STARTED_EVENT, bytes: 0
2016-08-31 16:22:15,372 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event CLIENT_REQUEST_STARTED_EVENT, bytes: 0
2016-08-31 16:22:15,373 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_REQUEST_STARTED_EVENT, bytes: 0
2016-08-31 16:23:40,083 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-08-31 16:23:40,083 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-08-31 16:23:40,084 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-08-31 16:23:40,084 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-08-31 16:23:40,084 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event TRANSFER_PART_COMPLETED_EVENT, bytes: 0
2016-08-31 16:24:42,754 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-08-31 16:24:42,754 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-08-31 16:24:42,755 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-08-31 16:24:42,755 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-08-31 16:24:42,755 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event TRANSFER_PART_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:08,954 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:08,954 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-08-31 16:25:08,954 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:08,955 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-08-31 16:25:08,955 [java-sdk-progress-listener-callback-thread] INFO  
scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) 
- Event TRANSFER_PART_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:09,327 [JUnit] INFO  contract.ContractTestUtils 
(ContractTestUtils.java:end(1365)) - Duration of Time to close() output stream: 
173,972,783,472 nS
2016-08-31 16:25:09,327 [JUnit] INFO  contract.ContractTestUtils 
(ContractTestUtils.java:end(1365)) - Duration of Time to write 256 MB in blocks 
of 65536: 174,530,704,251 nS
2016-08-31 16:25:09,328 [JUnit] INFO  scale.STestS3AHugeFileCreate 
(STestS3AHugeFileCreate.java:logFSState(232)) - File System state after 
operation:
S3AFileSystem{ inputPolicy=normal, partSize=104857600, 
enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, 
blockSize=33554432,