[jira] [Work logged] (GOBBLIN-1015) Support ability to write "native" Avro and Protobuf records into the Parquet Writer

Mon, 06 Jan 2020 17:21:05 -0800 


     [ 
https://issues.apache.org/jira/browse/GOBBLIN-1015?focusedWorklogId=367142&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367142
 ]

ASF GitHub Bot logged work on GOBBLIN-1015:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 07/Jan/20 01:19
            Start Date: 07/Jan/20 01:19
    Worklog Time Spent: 10m 
      Work Description: codecov-io commented on issue #2860: [GOBBLIN-1015] 
Support for direct Avro and Protobuf formats through Parquet writer
URL: 
https://github.com/apache/incubator-gobblin/pull/2860#issuecomment-571389021
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=h1)
 Report
   > Merging 
[#2860](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/d18783df7ab29b161cca59ca907fb47e1366b358?src=pr&el=desc)
 will **increase** coverage by `0.93%`.
   > The diff coverage is `33.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2860      +/-   ##
   ============================================
   + Coverage      44.8%   45.73%   +0.93%     
   - Complexity     8933     9103     +170     
   ============================================
     Files          1917     1917              
     Lines         72131    72168      +37     
     Branches       7956     7960       +4     
   ============================================
   + Hits          32316    33005     +689     
   + Misses        36832    36134     -698     
   - Partials       2983     3029      +46
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...g/apache/gobblin/writer/ParquetHdfsDataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi93cml0ZXIvUGFycXVldEhkZnNEYXRhV3JpdGVyLmphdmE=)
 | `100% <ø> (ø)` | `4 <0> (ø)` | :arrow_down: |
   | 
[...g/apache/gobblin/writer/ParquetHdfsDataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC1hcGFjaGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL1BhcnF1ZXRIZGZzRGF0YVdyaXRlci5qYXZh)
 | `100% <ø> (ø)` | `4 <0> (ø)` | :arrow_down: |
   | 
[.../main/java/org/apache/gobblin/test/TestRecord.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi10ZXN0LXV0aWxzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Rlc3QvVGVzdFJlY29yZC5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (?)` | |
   | 
[.../org/apache/gobblin/test/SequentialTestSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGVzdC9TZXF1ZW50aWFsVGVzdFNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...va/org/apache/gobblin/test/AnyToJsonConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGVzdC9BbnlUb0pzb25Db252ZXJ0ZXIuamF2YQ==)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../org/apache/gobblin/test/AnyToStringConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGVzdC9BbnlUb1N0cmluZ0NvbnZlcnRlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...pache/gobblin/writer/ParquetDataWriterBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC1hcGFjaGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL1BhcnF1ZXREYXRhV3JpdGVyQnVpbGRlci5qYXZh)
 | `64.51% <66.66%> (-21.7%)` | `4 <2> (-2)` | |
   | 
[...pache/gobblin/writer/ParquetDataWriterBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi93cml0ZXIvUGFycXVldERhdGFXcml0ZXJCdWlsZGVyLmphdmE=)
 | `64.51% <66.66%> (-21.7%)` | `4 <2> (-2)` | |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `61.42% <0%> (-1.43%)` | `4% <0%> (ø)` | |
   | 
[...src/main/java/org/apache/gobblin/runtime/Task.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFzay5qYXZh)
 | `67.05% <0%> (+0.23%)` | `82% <0%> (+1%)` | :arrow_up: |
   | ... and [46 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2860/diff?src=pr&el=tree-more)
 | |
   
   ------
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=footer).
 Last update 
[d18783d...c01cc58](https://codecov.io/gh/apache/incubator-gobblin/pull/2860?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


Issue Time Tracking
-------------------

            Worklog Id:     (was: 367142)
    Remaining Estimate: 0h
            Time Spent: 10m

> Support ability to write "native" Avro and Protobuf records into the Parquet 
> Writer
> -----------------------------------------------------------------------------------
>
>                 Key: GOBBLIN-1015
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1015
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Shirshanka Das
>            Assignee: Shirshanka Das
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> The current Parquet Writer requires a conversion to a Parquet Group object 
> before writing data to the Parquet file. 
> Since Parquet libraries natively support writing Avro and Protobuf in-mem 
> formats, we should enable that capability in Gobblin as well. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to