[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-02-11 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-777552648


   > No one should do this. Only want it for stable tests.
   
   that and to challenge sqllite for dominance in the embedded-database market
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-02-10 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776930067


   @ndimiduk this will let you run HBase against local FS now as if you turn 
off the checksumming the WAL will be synced properly...



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-02-09 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-776011799


   done the final comments and then rebased; unless yetus really blows up will 
merge in



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-02-08 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-775113631


   FWIW, here are the IOStats logged on the runs
   
   ABFS (which supports hsync)
   ```
   
.dfs.core.windows.net/stevel-testing/test/testSyncable?action=flush=false=2=true=90
   2021-02-08 12:27:48,642 INFO  [JUnit-testSyncable]: 
contract.AbstractContractCreateTest 
(AbstractContractCreateTest.java:validateSyncableSemantics(586)) - IOStatistics 
counters=((queue_shrunk_ops=1) (time_spent_task_wait.failures=0) 
(write_current_buffer_ops=2) (bytes_upload_failed=0) (bytes_upload=2) 
(time_spent_on_put_request.failures=0) (time_spent_on_put_request=2) 
(time_spent_task_wait=0) (bytes_upload_successfully=2));
   gauges=();
   minimums=((time_spent_task_wait.failures.min=-1) 
(time_spent_on_put_request.min=28) (time_spent_task_wait.min=-1) 
(time_spent_on_put_request.failures.min=-1));
   maximums=((time_spent_on_put_request.failures.max=-1) 
(time_spent_task_wait.max=-1) (time_spent_on_put_request.max=36) 
(time_spent_task_wait.failures.max=-1));
   means=((time_spent_task_wait.mean=(samples=0, sum=0, mean=0.)) 
(time_spent_on_put_request.failures.mean=(samples=0, sum=0, mean=0.)) 
(time_spent_task_wait.failures.mean=(samples=0, sum=0, mean=0.)) 
(time_spent_on_put_request.mean=(samples=2, sum=64, mean=32.)));
   ```
   
   s3a
   
   ```
   2021-02-08 12:29:05,294 [JUnit-testSyncable] WARN  
contract.AbstractContractCreateTest 
(AbstractContractCreateTest.java:validateSyncableSemantics(575)) - Output file 
was not created; this is an object store with different visibility semantics
   2021-02-08 12:29:05,500 [JUnit-testSyncable] INFO  
contract.AbstractContractCreateTest 
(AbstractContractCreateTest.java:validateSyncableSemantics(586)) - IOStatistics 
counters=((stream_write_bytes=2) (stream_write_queue_duration=0) 
(action_executor_acquired=1) (stream_write_block_uploads=1) 
(stream_write_exceptions=0) (stream_write_exceptions_completing_upload=0) 
(stream_write_total_time=0) (stream_write_total_data=2) 
(action_executor_acquired.failures=0));
   gauges=((stream_write_block_uploads_data_pending=0) 
(stream_write_block_uploads_pending=1));
   minimums=((action_executor_acquired.failures.min=-1) 
(action_executor_acquired.min=0));
   maximums=((action_executor_acquired.max=0) 
(action_executor_acquired.failures.max=-1));
   means=((action_executor_acquired.failures.mean=(samples=0, sum=0, 
mean=0.)) (action_executor_acquired.mean=(samples=1, sum=0, mean=0.)));
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-01-29 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-769827209


   checkstyles and whitespace. The javac changes are all from deprecating 
hflush capability



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-01-26 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-767785031


   checkstyle nits
   ```
   
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java:512:
  if (c == -1) {:24: Must have at least one statement. [EmptyBlock]
   
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:70:import
 org.apache.hadoop.util.StringUtils;:8: Unused import - 
org.apache.hadoop.util.StringUtils. [UnusedImports]
   
./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobAppendStream.java:30:import
 java.util.Locale;:8: Unused import - java.util.Locale. [UnusedImports]
   
./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SyncableDataOutputStream.java:31:import
 org.apache.hadoop.classification.InterfaceAudience;:1: Duplicate import to 
line 28 - org.apache.hadoop.classification.InterfaceAudience. [RedundantImport]
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-01-26 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-767784703


   > Steve had mentioned to me that he was thinking of HBase and us being able 
to more safely put WALs onto file:// with these changes (not having to just 
disable the checks with LocalFileSystem).
   
   you are still going to need some robust storage, RAID-1+ or similar, but we 
are getting sync all the way through, and you can query the streams to make 
sure they say they support it -including local fs. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2587: HADOOP-13327 Output Stream Specification.

2021-01-04 Thread GitBox 


steveloughran commented on pull request #2587:
URL: https://github.com/apache/hadoop/pull/2587#issuecomment-754016813


   * does not address final comments in that review
   * no ITest runs (I don't think it needs it; will look at again)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org