[jira] [Commented] (PARQUET-269) Upgrade parquet-scrooge to 3.19 or greater

2015-04-29 Thread Alex Levenson (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520249#comment-14520249 ] Alex Levenson commented on PARQUET-269: --- Both 3.17 and 3.18 had issues (3.17

[jira] [Commented] (PARQUET-268) Build is failing with parquet-scrooge errors.

2015-04-29 Thread Alex Levenson (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520236#comment-14520236 ] Alex Levenson commented on PARQUET-268: --- Looks like you beat me to it, I was doing

Re: Doing the Spark shuffle on Parquet floors

2015-04-29 Thread Matt Massie
I guess that the correct answer is it depends :) Luckily, all the write path costs are being shouldered by the mappers. The dictionary encoding will almost certainly decrease the memory/gc pressure we're seeing on the reducers. Without this shuffle manager, Spark serializes all intermediate data

Re: Doing the Spark shuffle on Parquet floors

2015-04-29 Thread Dmitriy Ryaboy
Matt, this is good stuff! :). On Wed, Apr 29, 2015 at 4:28 PM, Matt Massie mas...@berkeley.edu wrote: I guess that the correct answer is it depends :) Luckily, all the write path costs are being shouldered by the mappers. The dictionary encoding will almost certainly decrease the memory/gc

[jira] [Commented] (PARQUET-268) Build is failing with parquet-scrooge errors.

2015-04-29 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520532#comment-14520532 ] Ryan Blue commented on PARQUET-268: --- I'm going to do the downgrade and ignore the

[jira] [Commented] (PARQUET-268) Build is failing with parquet-scrooge errors.

2015-04-29 Thread Alex Levenson (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520515#comment-14520515 ] Alex Levenson commented on PARQUET-268: --- Looks like downgrading won't work for us.

[jira] [Assigned] (PARQUET-47) SERDE backed schema for parquet storage in Hive

2015-04-29 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh reassigned PARQUET-47: - Assignee: Ashish K Singh SERDE backed schema for parquet storage in Hive

Re: Doing the Spark shuffle on Parquet floors

2015-04-29 Thread Julien Le Dem
Matt: I'm looking forward to the blog post that goes with this :) On Wed, Apr 29, 2015 at 4:36 PM, Dmitriy Ryaboy dvrya...@gmail.com wrote: Matt, this is good stuff! :). On Wed, Apr 29, 2015 at 4:28 PM, Matt Massie mas...@berkeley.edu wrote: I guess that the correct answer is it depends :)

[jira] [Updated] (PARQUET-269) Upgrade parquet-scrooge to 3.18.1 or greater

2015-04-29 Thread Alex Levenson (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Levenson updated PARQUET-269: -- Summary: Upgrade parquet-scrooge to 3.18.1 or greater (was: Upgrade parquet-scrooge to 3.19

[jira] [Resolved] (PARQUET-268) Build is failing with parquet-scrooge errors.

2015-04-29 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved PARQUET-268. --- Resolution: Fixed Assignee: Ryan Blue Build is failing with parquet-scrooge errors.

[jira] [Assigned] (PARQUET-270) Add legend to parquet-tools readme.md

2015-04-29 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue reassigned PARQUET-270: - Assignee: Ryan Blue Add legend to parquet-tools readme.md