[
https://issues.apache.org/jira/browse/BEAM-214?focusedWorklogId=82006&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82006
]
ASF GitHub Bot logged work on BEAM-214:
---------------------------------------
Author: ASF GitHub Bot
Created on: 19/Mar/18 20:24
Start Date: 19/Mar/18 20:24
Worklog Time Spent: 10m
Work Description: Igosuki commented on issue #1851: [BEAM-214] Add
ParquetIO
URL: https://github.com/apache/beam/pull/1851#issuecomment-374360899
@jbonofre I did when you started it, once to create a Druid segment
rewriter and another to test performance of a simple windowed sum over a
large number of keys vs Spark.
My files go from small files (due to people having implemented constant
partitionning with different sized keys) to 400Mo parquet files.
I tested your code vs avro io vs the recent Hadoop input split over a few
TBs.
In the end, I had to fix coders which catch EOF because Flink throws EOF to
trigger spill on disk and reading parquet quickly overflows the heap...
The code works fine, but Hadoop input format io over S3A pefrozm better. I
think we need some examples to attain the same performance as Spark group
by + partition when there is a large cardinality over keys reading binary
formats.
On Mon, Mar 19, 2018, 6:20 PM Jean-Baptiste Onofré <[email protected]>
wrote:
> @lgajowy <https://github.com/lgajowy> by the way, does it mean that you
> already used the first "stage" of this PR ? I'm curious ;)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/1851#issuecomment-374294586>, or mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AAYyhh-hh3qzHIlJ0PNCZcFgTHr02wRPks5tf-jZgaJpZM4LupIS>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 82006)
Time Spent: 3h 40m (was: 3.5h)
> Create Parquet IO
> -----------------
>
> Key: BEAM-214
> URL: https://issues.apache.org/jira/browse/BEAM-214
> Project: Beam
> Issue Type: Improvement
> Components: io-ideas
> Reporter: Neville Li
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Would be nice to support Parquet files with projection and predicates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
