[jira] [Commented] (HUDI-677) Abstract/Refactor all transaction management logic into a set of classes from HoodieWriteClient

2020-03-28 Thread hong dongdong (Jira)


[ 
https://issues.apache.org/jira/browse/HUDI-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070217#comment-17070217
 ] 

hong dongdong commented on HUDI-677:


[~vinoth] Of course not. 

> Abstract/Refactor all transaction management logic into a set of classes from 
> HoodieWriteClient
> ---
>
> Key: HUDI-677
> URL: https://issues.apache.org/jira/browse/HUDI-677
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
> Fix For: 0.6.0
>
>
> Over time a lot of the core transaction management code has been  split 
> across various files in hudi-client.. We want to clean this up and present a 
> nice interface.. 
> Some notes and thoughts and suggestions..  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free

2020-03-28 Thread GitBox
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark 
free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=h1) 
Report
   > Merging 
[#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33=desc)
 will **decrease** coverage by `0.06%`.
   > The diff coverage is `97.36%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-hudi/pull/1460/graphs/tree.svg?width=650=150=pr=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#1460  +/-   ##
   
   - Coverage 67.66%   67.60%   -0.07% 
 Complexity  261  261  
   
 Files   342  348   +6 
 Lines 1651016670 +160 
 Branches   1684 1693   +9 
   
   + Hits  1117211270  +98 
   - Misses 4599 4661  +62 
 Partials739  739  
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
   | 
[...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=)
 | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=)
 | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==)
 | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==)
 | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
   | 
[...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh)
 | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | ... and [22 
more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=footer).
 Last update 

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free

2020-03-28 Thread GitBox
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark 
free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=h1) 
Report
   > Merging 
[#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33=desc)
 will **decrease** coverage by `0.08%`.
   > The diff coverage is `97.36%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-hudi/pull/1460/graphs/tree.svg?width=650=150=pr=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#1460  +/-   ##
   
   - Coverage 67.66%   67.58%   -0.09% 
 Complexity  261  261  
   
 Files   342  348   +6 
 Lines 1651016670 +160 
 Branches   1684 1693   +9 
   
   + Hits  1117211266  +94 
   - Misses 4599 4665  +66 
 Partials739  739  
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
   | 
[...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=)
 | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=)
 | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==)
 | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==)
 | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
   | 
[...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh)
 | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | ... and [22 
more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=footer).
 Last update 

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free

2020-03-28 Thread GitBox
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark 
free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=h1) 
Report
   > Merging 
[#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33=desc)
 will **decrease** coverage by `0.08%`.
   > The diff coverage is `97.36%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-hudi/pull/1460/graphs/tree.svg?width=650=150=pr=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#1460  +/-   ##
   
   - Coverage 67.66%   67.58%   -0.09% 
 Complexity  261  261  
   
 Files   342  348   +6 
 Lines 1651016670 +160 
 Branches   1684 1693   +9 
   
   + Hits  1117211266  +94 
   - Misses 4599 4665  +66 
 Partials739  739  
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
   | 
[...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=)
 | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=)
 | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==)
 | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==)
 | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
   | 
[...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh)
 | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | ... and [22 
more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=footer).
 Last update 

[GitHub] [incubator-hudi] codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark free

2020-03-28 Thread GitBox
codecov-io edited a comment on issue #1460: [HUDI-679] Make io package Spark 
free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605428358
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=h1) 
Report
   > Merging 
[#1460](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-hudi/commit/1713f686f86e8c2f0a908c313cca9b595c6aed33=desc)
 will **decrease** coverage by `0.06%`.
   > The diff coverage is `97.36%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-hudi/pull/1460/graphs/tree.svg?width=650=150=pr=VTTXabwbs2)](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#1460  +/-   ##
   
   - Coverage 67.66%   67.60%   -0.07% 
 Complexity  261  261  
   
 Files   342  348   +6 
 Lines 1651016670 +160 
 Branches   1684 1693   +9 
   
   + Hits  1117211270  +98 
   - Misses 4599 4661  +62 
 Partials739  739  
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[.../hudi/execution/MergeOnReadLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL01lcmdlT25SZWFkTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `64.70% <66.66%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...g/apache/hudi/client/SparkTaskContextSupplier.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrVGFza0NvbnRleHRTdXBwbGllci5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (?)` | |
   | 
[...g/apache/hudi/execution/BulkInsertMapFunction.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0J1bGtJbnNlcnRNYXBGdW5jdGlvbi5qYXZh)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[.../hudi/execution/CopyOnWriteLazyInsertIterable.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL0NvcHlPbldyaXRlTGF6eUluc2VydEl0ZXJhYmxlLmphdmE=)
 | `80.76% <100.00%> (+0.37%)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQXBwZW5kSGFuZGxlLmphdmE=)
 | `84.17% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...in/java/org/apache/hudi/io/HoodieCreateHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllQ3JlYXRlSGFuZGxlLmphdmE=)
 | `84.61% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllTWVyZ2VIYW5kbGUuamF2YQ==)
 | `79.31% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...ain/java/org/apache/hudi/io/HoodieWriteHandle.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vSG9vZGllV3JpdGVIYW5kbGUuamF2YQ==)
 | `75.00% <100.00%> (+1.66%)` | `0.00 <0.00> (ø)` | |
   | 
[...rg/apache/hudi/io/storage/HoodieParquetWriter.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVQYXJxdWV0V3JpdGVyLmphdmE=)
 | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | 
[...he/hudi/io/storage/HoodieStorageWriterFactory.java](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVTdG9yYWdlV3JpdGVyRmFjdG9yeS5qYXZh)
 | `93.75% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | ... and [22 
more](https://codecov.io/gh/apache/incubator-hudi/pull/1460/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-hudi/pull/1460?src=pr=footer).
 Last update 

[GitHub] [incubator-hudi] leesf commented on issue #1460: [HUDI-679] Make io package Spark free

2020-03-28 Thread GitBox
leesf commented on issue #1460: [HUDI-679] Make io package Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#issuecomment-605560443
 
 
   @yanghua @vinothchandar Thanks for your review, just updated the PR to 
address your comments. PTAL.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


Build failed in Jenkins: hudi-snapshot-deployment-0.5 #231

2020-03-28 Thread Apache Jenkins Server
See 


Changes:


--
[...truncated 2.40 KB...]
/home/jenkins/tools/maven/apache-maven-3.5.4/conf:
logging
settings.xml
toolchains.xml

/home/jenkins/tools/maven/apache-maven-3.5.4/conf/logging:
simplelogger.properties

/home/jenkins/tools/maven/apache-maven-3.5.4/lib:
aopalliance-1.0.jar
cdi-api-1.0.jar
cdi-api.license
commons-cli-1.4.jar
commons-cli.license
commons-io-2.5.jar
commons-io.license
commons-lang3-3.5.jar
commons-lang3.license
ext
guava-20.0.jar
guice-4.2.0-no_aop.jar
jansi-1.17.1.jar
jansi-native
javax.inject-1.jar
jcl-over-slf4j-1.7.25.jar
jcl-over-slf4j.license
jsr250-api-1.0.jar
jsr250-api.license
maven-artifact-3.5.4.jar
maven-artifact.license
maven-builder-support-3.5.4.jar
maven-builder-support.license
maven-compat-3.5.4.jar
maven-compat.license
maven-core-3.5.4.jar
maven-core.license
maven-embedder-3.5.4.jar
maven-embedder.license
maven-model-3.5.4.jar
maven-model-builder-3.5.4.jar
maven-model-builder.license
maven-model.license
maven-plugin-api-3.5.4.jar
maven-plugin-api.license
maven-repository-metadata-3.5.4.jar
maven-repository-metadata.license
maven-resolver-api-1.1.1.jar
maven-resolver-api.license
maven-resolver-connector-basic-1.1.1.jar
maven-resolver-connector-basic.license
maven-resolver-impl-1.1.1.jar
maven-resolver-impl.license
maven-resolver-provider-3.5.4.jar
maven-resolver-provider.license
maven-resolver-spi-1.1.1.jar
maven-resolver-spi.license
maven-resolver-transport-wagon-1.1.1.jar
maven-resolver-transport-wagon.license
maven-resolver-util-1.1.1.jar
maven-resolver-util.license
maven-settings-3.5.4.jar
maven-settings-builder-3.5.4.jar
maven-settings-builder.license
maven-settings.license
maven-shared-utils-3.2.1.jar
maven-shared-utils.license
maven-slf4j-provider-3.5.4.jar
maven-slf4j-provider.license
org.eclipse.sisu.inject-0.3.3.jar
org.eclipse.sisu.inject.license
org.eclipse.sisu.plexus-0.3.3.jar
org.eclipse.sisu.plexus.license
plexus-cipher-1.7.jar
plexus-cipher.license
plexus-component-annotations-1.7.1.jar
plexus-component-annotations.license
plexus-interpolation-1.24.jar
plexus-interpolation.license
plexus-sec-dispatcher-1.4.jar
plexus-sec-dispatcher.license
plexus-utils-3.1.0.jar
plexus-utils.license
slf4j-api-1.7.25.jar
slf4j-api.license
wagon-file-3.1.0.jar
wagon-file.license
wagon-http-3.1.0-shaded.jar
wagon-http.license
wagon-provider-api-3.1.0.jar
wagon-provider-api.license

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/ext:
README.txt

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native:
freebsd32
freebsd64
linux32
linux64
osx
README.txt
windows32
windows64

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/osx:
libjansi.jnilib

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows32:
jansi.dll

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows64:
jansi.dll
Finished /home/jenkins/tools/maven/apache-maven-3.5.4 Directory Listing :
Detected current version as: 
'HUDI_home=
0.6.0-SNAPSHOT'
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-spark_2.11:jar:0.6.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.hudi:hudi-spark_${scala.binary.version}:[unknown-version], 

 line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-timeline-service:jar:0.6.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found 
duplicate declaration of plugin org.jacoco:jacoco-maven-plugin @ 
org.apache.hudi:hudi-timeline-service:[unknown-version], 

 line 58, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-utilities_2.11:jar:0.6.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.hudi:hudi-utilities_${scala.binary.version}:[unknown-version], 

 line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-spark-bundle_2.11:jar:0.6.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ 

[incubator-hudi] branch asf-site updated: Travis CI build asf-site

2020-03-28 Thread vinoth
This is an automated email from the ASF dual-hosted git repository.

vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 96f3d74  Travis CI build asf-site
96f3d74 is described below

commit 96f3d746d491c06e67c494985452fd95d0b831ee
Author: CI 
AuthorDate: Sun Mar 29 03:30:47 2020 +

Travis CI build asf-site
---
 test-content/assets/js/lunr/lunr-store.js |   4 +-
 test-content/cn/docs/0.5.2-querying_data.html | 102 +-
 test-content/cn/docs/querying_data.html   | 102 +-
 test-content/docs/0.5.2-querying_data.html|   3 +-
 test-content/docs/querying_data.html  |   3 +-
 5 files changed, 204 insertions(+), 10 deletions(-)

diff --git a/test-content/assets/js/lunr/lunr-store.js 
b/test-content/assets/js/lunr/lunr-store.js
index 1077cf3..9351ab6 100644
--- a/test-content/assets/js/lunr/lunr-store.js
+++ b/test-content/assets/js/lunr/lunr-store.js
@@ -435,7 +435,7 @@ var store = [{
 "url": "https://hudi.apache.org/docs/0.5.2-writing_data.html;,
 "teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
 "title": "查询 Hudi 数据集",
-"excerpt":"从概念上讲,Hudi物理存储一次数据到DFS上,同时在其上提供三个逻辑视图,如之前所述。 数据集同步到Hive 
Metastore后,它将提供由Hudi的自定义输入格式支持的Hive外部表。一旦提供了适当的Hudi捆绑包, 
就可以通过Hive、Spark和Presto之类的常用查询引擎来查询数据集。 具体来说,在写入过程中传递了两个由table name命名的Hive表。 
例如,如果table name = hudi_tbl,我们得到 hudi_tbl 实现了由 HoodieParquetInputFormat 
支持的数据集的读优化视图,从而提供了纯列式数据。 hudi_tbl_rt 实现了由 HoodieParquetRealtimeInputFormat 
支持的数据集的实时视图,从而提供了基础数据和日志数据的合并视图。 如概念部分所述,增量处理所需要的 
一个关键原语是增量拉取(以从数据集中获取更改流/日志)。您可以增量提取Hudi数据集,这意味着自指定的即时时间起, 您可�
 �只获得全部更新和新行。 这与插入更新一起使用,对于构建某 [...]
+"excerpt":"从概念上讲,Hudi物理存储一次数据到DFS上,同时在其上提供三个逻辑视图,如之前所述。 数据集同步到Hive 
Metastore后,它将提供由Hudi的自定义输入格式支持的Hive外部表。一旦提供了适当的Hudi捆绑包, 
就可以通过Hive、Spark和Presto之类的常用查询引擎来查询数据集。 具体来说,在写入过程中传递了两个由table name命名的Hive表。 
例如,如果table name = hudi_tbl,我们得到 hudi_tbl 实现了由 HoodieParquetInputFormat 
支持的数据集的读优化视图,从而提供了纯列式数据。 hudi_tbl_rt 实现了由 HoodieParquetRealtimeInputFormat 
支持的数据集的实时视图,从而提供了基础数据和日志数据的合并视图。 如概念部分所述,增量处理所需要的 
一个关键原语是增量拉取(以从数据集中获取更改流/日志)。您可以增量提取Hudi数据集,这意味着自指定的即时时间起, 您可�
 �只获得全部更新和新行。 这与插入更新一起使用,对于构建某 [...]
 "tags": [],
 "url": "https://hudi.apache.org/cn/docs/0.5.2-querying_data.html;,
 "teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
@@ -600,7 +600,7 @@ var store = [{
 "url": "https://hudi.apache.org/docs/writing_data.html;,
 "teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
 "title": "查询 Hudi 数据集",
-"excerpt":"从概念上讲,Hudi物理存储一次数据到DFS上,同时在其上提供三个逻辑视图,如之前所述。 数据集同步到Hive 
Metastore后,它将提供由Hudi的自定义输入格式支持的Hive外部表。一旦提供了适当的Hudi捆绑包, 
就可以通过Hive、Spark和Presto之类的常用查询引擎来查询数据集。 具体来说,在写入过程中传递了两个由table name命名的Hive表。 
例如,如果table name = hudi_tbl,我们得到 hudi_tbl 实现了由 HoodieParquetInputFormat 
支持的数据集的读优化视图,从而提供了纯列式数据。 hudi_tbl_rt 实现了由 HoodieParquetRealtimeInputFormat 
支持的数据集的实时视图,从而提供了基础数据和日志数据的合并视图。 如概念部分所述,增量处理所需要的 
一个关键原语是增量拉取(以从数据集中获取更改流/日志)。您可以增量提取Hudi数据集,这意味着自指定的即时时间起, 您可�
 �只获得全部更新和新行。 这与插入更新一起使用,对于构建某 [...]
+"excerpt":"从概念上讲,Hudi物理存储一次数据到DFS上,同时在其上提供三个逻辑视图,如之前所述。 数据集同步到Hive 
Metastore后,它将提供由Hudi的自定义输入格式支持的Hive外部表。一旦提供了适当的Hudi捆绑包, 
就可以通过Hive、Spark和Presto之类的常用查询引擎来查询数据集。 具体来说,在写入过程中传递了两个由table name命名的Hive表。 
例如,如果table name = hudi_tbl,我们得到 hudi_tbl 实现了由 HoodieParquetInputFormat 
支持的数据集的读优化视图,从而提供了纯列式数据。 hudi_tbl_rt 实现了由 HoodieParquetRealtimeInputFormat 
支持的数据集的实时视图,从而提供了基础数据和日志数据的合并视图。 如概念部分所述,增量处理所需要的 
一个关键原语是增量拉取(以从数据集中获取更改流/日志)。您可以增量提取Hudi数据集,这意味着自指定的即时时间起, 您可�
 �只获得全部更新和新行。 这与插入更新一起使用,对于构建某 [...]
 "tags": [],
 "url": "https://hudi.apache.org/cn/docs/querying_data.html;,
 "teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
diff --git a/test-content/cn/docs/0.5.2-querying_data.html 
b/test-content/cn/docs/0.5.2-querying_data.html
index 0f4a441..5d337d6 100644
--- a/test-content/cn/docs/0.5.2-querying_data.html
+++ b/test-content/cn/docs/0.5.2-querying_data.html
@@ -335,6 +335,12 @@
   
  IN 
THIS PAGE
 
+  查询引擎支持列表
+
+  读优化表
+  实时表
+
+  
   Hive
 
   读优化表
@@ -352,7 +358,7 @@
   Presto
   Impala(此功能还未正式发布)
 
-  读优化表
+  读优化表
 
   
 
@@ -377,6 +383,94 @@
 并与其他表(数据集/维度)结合以写出增量到目标Hudi数据集。增量视图是通过查询上表之一实现的,并具有特殊配置,
 该特殊配置指示查询计划仅需要从数据集中获取增量数据。
 
+查询引擎支持列表
+
+下面的表格展示了各查询引擎是否支持Hudi格式
+
+读优化表
+
+
+  
+
+  查询引擎
+  实时视图
+  增量拉取
+
+  
+  
+
+  Hive
+  Y
+  Y
+
+
+  Spark SQL
+  Y
+  Y
+
+
+  Spark Datasource
+  Y
+  Y
+
+
+  Presto
+  Y
+  N
+
+
+  Impala
+  Y
+  N
+
+  
+
+
+实时表
+
+
+  
+
+  查询引擎
+  实时视图
+  增量拉取
+  读优化表
+
+  
+  
+
+  Hive
+  Y
+  Y
+  Y
+
+
+  Spark SQL
+  Y
+  Y
+  Y
+
+

[GitHub] [incubator-hudi] vinothchandar commented on issue #1421: [HUDI-724] Parallelize getSmallFiles for partitions

2020-03-28 Thread GitBox
vinothchandar commented on issue #1421: [HUDI-724] Parallelize getSmallFiles 
for partitions
URL: https://github.com/apache/incubator-hudi/pull/1421#issuecomment-605552930
 
 
   Took a pass. LGTM overall
   since @bvaradar is the assignee.. his call :) 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[incubator-hudi] branch asf-site updated: [MINOR] Update doc to include inc query on partitions (#1454)

2020-03-28 Thread leesf
This is an automated email from the ASF dual-hosted git repository.

leesf pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 18ce570  [MINOR] Update doc to include inc query on partitions (#1454)
18ce570 is described below

commit 18ce5708e073e80779f6dcc00d388b4cb0cc758a
Author: YanJia-Gary-Li 
AuthorDate: Sat Mar 28 20:28:48 2020 -0700

[MINOR] Update doc to include inc query on partitions (#1454)
---
 docs/_docs/0.5.2/2_3_querying_data.cn.md | 31 ++-
 docs/_docs/0.5.2/2_3_querying_data.md|  3 ++-
 docs/_docs/2_3_querying_data.cn.md   | 31 ++-
 docs/_docs/2_3_querying_data.md  |  3 ++-
 4 files changed, 64 insertions(+), 4 deletions(-)

diff --git a/docs/_docs/0.5.2/2_3_querying_data.cn.md 
b/docs/_docs/0.5.2/2_3_querying_data.cn.md
index 74afcef..77ad2d7 100644
--- a/docs/_docs/0.5.2/2_3_querying_data.cn.md
+++ b/docs/_docs/0.5.2/2_3_querying_data.cn.md
@@ -25,6 +25,33 @@ language: cn
 
并与其他表(数据集/维度)结合以[写出增量](/cn/docs/0.5.2-writing_data.html)到目标Hudi数据集。增量视图是通过查询上表之一实现的,并具有特殊配置,
 该特殊配置指示查询计划仅需要从数据集中获取增量数据。
 
+
+## 查询引擎支持列表
+
+下面的表格展示了各查询引擎是否支持Hudi格式
+
+### 读优化表
+  
+|查询引擎|实时视图|增量拉取|
+|||---|
+|**Hive**|Y|Y|
+|**Spark SQL**|Y|Y|
+|**Spark Datasource**|Y|Y|
+|**Presto**|Y|N|
+|**Impala**|Y|N|
+
+
+### 实时表
+
+|查询引擎|实时视图|增量拉取|读优化表|
+|||---|--|
+|**Hive**|Y|Y|Y|
+|**Spark SQL**|Y|Y|Y|
+|**Spark Datasource**|N|N|Y|
+|**Presto**|N|N|Y|
+|**Impala**|N|N|Y|
+
+
 接下来,我们将详细讨论在每个查询引擎上如何访问所有三个视图。
 
 ## Hive
@@ -128,7 +155,9 @@ scala> sqlContext.sql("select count(*) from hudi_rt where 
datestr = '2016-10-02'
  DataSourceReadOptions.VIEW_TYPE_INCREMENTAL_OPT_VAL())
  .option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY(),
 )
- .load(tablePath); // For incremental view, pass in the root/base path of 
dataset
+ .option(DataSourceReadOptions.INCR_PATH_GLOB_OPT_KEY(),
+"/year=2020/month=*/day=*") // 可选,从指定的分区增量拉取
+ .load(tablePath); // 用数据集的最底层路径
 ```
 
 请参阅[设置](/cn/docs/0.5.2-configurations.html#spark-datasource)部分,以查看所有数据源选项。
diff --git a/docs/_docs/0.5.2/2_3_querying_data.md 
b/docs/_docs/0.5.2/2_3_querying_data.md
index 0c28b12..9d17e72 100644
--- a/docs/_docs/0.5.2/2_3_querying_data.md
+++ b/docs/_docs/0.5.2/2_3_querying_data.md
@@ -55,7 +55,7 @@ Note that `Read Optimized` queries are not applicable for 
COPY_ON_WRITE tables.
 |**Spark SQL**|Y|Y|Y|
 |**Spark Datasource**|N|N|Y|
 |**Presto**|N|N|Y|
-|**Impala**|N|N|N|
+|**Impala**|N|N|Y|
 
 
 In sections, below we will discuss specific setup to access different query 
types from different query engines. 
@@ -148,6 +148,7 @@ The following snippet shows how to obtain all records 
changed after `beginInstan
  .format("org.apache.hudi")
  .option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY(), 
DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
  .option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY(), 
)
+ .option(DataSourceReadOptions.INCR_PATH_GLOB_OPT_KEY(), 
"/year=2020/month=*/day=*") // Optional, use glob pattern if querying certain 
partitions
  .load(tablePath); // For incremental query, pass in the root/base path of 
table
  
 hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
diff --git a/docs/_docs/2_3_querying_data.cn.md 
b/docs/_docs/2_3_querying_data.cn.md
index b2c4870..1fa91d1 100644
--- a/docs/_docs/2_3_querying_data.cn.md
+++ b/docs/_docs/2_3_querying_data.cn.md
@@ -24,6 +24,33 @@ language: cn
 
并与其他表(数据集/维度)结合以[写出增量](/cn/docs/writing_data.html)到目标Hudi数据集。增量视图是通过查询上表之一实现的,并具有特殊配置,
 该特殊配置指示查询计划仅需要从数据集中获取增量数据。
 
+
+## 查询引擎支持列表
+
+下面的表格展示了各查询引擎是否支持Hudi格式
+
+### 读优化表
+  
+|查询引擎|实时视图|增量拉取|
+|||---|
+|**Hive**|Y|Y|
+|**Spark SQL**|Y|Y|
+|**Spark Datasource**|Y|Y|
+|**Presto**|Y|N|
+|**Impala**|Y|N|
+
+
+### 实时表
+
+|查询引擎|实时视图|增量拉取|读优化表|
+|||---|--|
+|**Hive**|Y|Y|Y|
+|**Spark SQL**|Y|Y|Y|
+|**Spark Datasource**|N|N|Y|
+|**Presto**|N|N|Y|
+|**Impala**|N|N|Y|
+
+
 接下来,我们将详细讨论在每个查询引擎上如何访问所有三个视图。
 
 ## Hive
@@ -127,7 +154,9 @@ scala> sqlContext.sql("select count(*) from hudi_rt where 
datestr = '2016-10-02'
  DataSourceReadOptions.VIEW_TYPE_INCREMENTAL_OPT_VAL())
  .option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY(),
 )
- .load(tablePath); // For incremental view, pass in the root/base path of 
dataset
+ .option(DataSourceReadOptions.INCR_PATH_GLOB_OPT_KEY(),
+"/year=2020/month=*/day=*") // 可选,从指定的分区增量拉取
+ .load(tablePath); // 用数据集的最底层路径
 ```
 
 请参阅[设置](/cn/docs/configurations.html#spark-datasource)部分,以查看所有数据源选项。
diff --git a/docs/_docs/2_3_querying_data.md b/docs/_docs/2_3_querying_data.md
index 875b7f0..3e6a436 100644
--- a/docs/_docs/2_3_querying_data.md
+++ 

[GitHub] [incubator-hudi] leesf merged pull request #1454: MINOR update doc to include inc query on partitions

2020-03-28 Thread GitBox
leesf merged pull request #1454: MINOR update doc to include inc query on 
partitions
URL: https://github.com/apache/incubator-hudi/pull/1454
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-hudi] vinothchandar commented on issue #1455: [SUPPORT] Hudi upsert run into exception: java.lang.NoSuchMethodError: java.lang.Math.floorMod(JI)I

2020-03-28 Thread GitBox
vinothchandar commented on issue #1455: [SUPPORT] Hudi upsert run into 
exception:  java.lang.NoSuchMethodError: java.lang.Math.floorMod(JI)I
URL: https://github.com/apache/incubator-hudi/issues/1455#issuecomment-605552838
 
 
   Thanks @EdwinGuo and @lamber-ken .. Please raise a JIRA if there is follow 
up work and clsoe this issue :) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-hudi] vinothchandar commented on issue #1453: HUDI-644 kafka connect checkpoint provider

2020-03-28 Thread GitBox
vinothchandar commented on issue #1453: HUDI-644 kafka connect checkpoint 
provider
URL: https://github.com/apache/incubator-hudi/pull/1453#issuecomment-605552741
 
 
   Slightly behind.. Will chime in here soon :) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Assigned] (HUDI-677) Abstract/Refactor all transaction management logic into a set of classes from HoodieWriteClient

2020-03-28 Thread Vinoth Chandar (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinoth Chandar reassigned HUDI-677:
---

Assignee: Vinoth Chandar  (was: hong dongdong)

> Abstract/Refactor all transaction management logic into a set of classes from 
> HoodieWriteClient
> ---
>
> Key: HUDI-677
> URL: https://issues.apache.org/jira/browse/HUDI-677
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
> Fix For: 0.6.0
>
>
> Over time a lot of the core transaction management code has been  split 
> across various files in hudi-client.. We want to clean this up and present a 
> nice interface.. 
> Some notes and thoughts and suggestions..  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HUDI-677) Abstract/Refactor all transaction management logic into a set of classes from HoodieWriteClient

2020-03-28 Thread Vinoth Chandar (Jira)


[ 
https://issues.apache.org/jira/browse/HUDI-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070195#comment-17070195
 ] 

Vinoth Chandar commented on HUDI-677:
-

Let me take this over..  if you don't mind [~hongdongdong] .. we can discuss on 
the PR.. 

> Abstract/Refactor all transaction management logic into a set of classes from 
> HoodieWriteClient
> ---
>
> Key: HUDI-677
> URL: https://issues.apache.org/jira/browse/HUDI-677
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: hong dongdong
>Priority: Major
> Fix For: 0.6.0
>
>
> Over time a lot of the core transaction management code has been  split 
> across various files in hudi-client.. We want to clean this up and present a 
> nice interface.. 
> Some notes and thoughts and suggestions..  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1460: [HUDI-679] Make io package Spark free

2020-03-28 Thread GitBox
yanghua commented on a change in pull request #1460: [HUDI-679] Make io package 
Spark free
URL: https://github.com/apache/incubator-hudi/pull/1460#discussion_r399737990
 
 

 ##
 File path: 
hudi-client/src/main/java/org/apache/hudi/client/SparkTaskContextDetailSupplier.java
 ##
 @@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.spark.TaskContext;
+
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * Spark Supplier.
+ */
+public interface SparkTaskContextDetailSupplier extends Supplier, 
Serializable {
 
 Review comment:
   Just online seeing the latest changes comes from @leesf . Yes, it seems this 
is a better abstraction.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-hudi] hddong commented on a change in pull request #1452: [HUDI-740]Fix can not specify the sparkMaster of cleans run command

2020-03-28 Thread GitBox
hddong commented on a change in pull request #1452: [HUDI-740]Fix can not 
specify the sparkMaster of cleans run command
URL: https://github.com/apache/incubator-hudi/pull/1452#discussion_r399733435
 
 

 ##
 File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java
 ##
 @@ -62,7 +63,9 @@ public static void main(String[] args) throws Exception {
 
 SparkCommand cmd = SparkCommand.valueOf(command);
 
-JavaSparkContext jsc = SparkUtil.initJavaSparkConf("hoodie-cli-" + 
command);
+JavaSparkContext jsc = cmd == SparkCommand.CLEAN
 
 Review comment:
   @prashantwason my mistake, thanks for your point out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (HUDI-736) Simplify ReflectionUtils#getTopLevelClasses

2020-03-28 Thread Suneel Marthi (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suneel Marthi updated HUDI-736:
---
Status: Open  (was: New)

> Simplify ReflectionUtils#getTopLevelClasses 
> 
>
> Key: HUDI-736
> URL: https://issues.apache.org/jira/browse/HUDI-736
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (HUDI-736) Simplify ReflectionUtils#getTopLevelClasses

2020-03-28 Thread Suneel Marthi (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suneel Marthi updated HUDI-736:
---
Fix Version/s: 0.6.0

> Simplify ReflectionUtils#getTopLevelClasses 
> 
>
> Key: HUDI-736
> URL: https://issues.apache.org/jira/browse/HUDI-736
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
> Fix For: 0.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (HUDI-736) Simplify ReflectionUtils#getTopLevelClasses

2020-03-28 Thread Suneel Marthi (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suneel Marthi reassigned HUDI-736:
--

Assignee: Suneel Marthi

> Simplify ReflectionUtils#getTopLevelClasses 
> 
>
> Key: HUDI-736
> URL: https://issues.apache.org/jira/browse/HUDI-736
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HUDI-737) Simplify/Eliminate need for CollectionUtils#Maps/MapsBuilder

2020-03-28 Thread Suneel Marthi (Jira)


[ 
https://issues.apache.org/jira/browse/HUDI-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070158#comment-17070158
 ] 

Suneel Marthi commented on HUDI-737:


Fixed as part of HUDI-479

> Simplify/Eliminate need for CollectionUtils#Maps/MapsBuilder
> 
>
> Key: HUDI-737
> URL: https://issues.apache.org/jira/browse/HUDI-737
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
> Fix For: 0.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (HUDI-479) Eliminate use of guava if possible

2020-03-28 Thread Suneel Marthi (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suneel Marthi resolved HUDI-479.

Resolution: Fixed

> Eliminate use of guava if possible
> --
>
> Key: HUDI-479
> URL: https://issues.apache.org/jira/browse/HUDI-479
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.6.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (HUDI-479) Eliminate use of guava if possible

2020-03-28 Thread Suneel Marthi (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suneel Marthi closed HUDI-479.
--

> Eliminate use of guava if possible
> --
>
> Key: HUDI-479
> URL: https://issues.apache.org/jira/browse/HUDI-479
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.6.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Reopened] (HUDI-479) Eliminate use of guava if possible

2020-03-28 Thread Suneel Marthi (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suneel Marthi reopened HUDI-479:


> Eliminate use of guava if possible
> --
>
> Key: HUDI-479
> URL: https://issues.apache.org/jira/browse/HUDI-479
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Suneel Marthi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.6.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (HUDI-744) Redistribute files in hudi-common utils package

2020-03-28 Thread Vinoth Chandar (Jira)


 [ 
https://issues.apache.org/jira/browse/HUDI-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinoth Chandar updated HUDI-744:

Status: Open  (was: New)