[jira] [Updated] (ARROW-2369) Large (>~20 GB) files written to Parquet via PyArrow are corrupted

2018-03-30 Thread Justin Tan (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tan updated ARROW-2369: -- Description: When writing large Parquet files (above 20 GB or so) from Pandas to Parquet via the comman

[jira] [Created] (ARROW-2369) Large (>~20 GB) files written to Parquet via PyArrow are corrupted

2018-03-30 Thread Justin Tan (JIRA)
Justin Tan created ARROW-2369: - Summary: Large (>~20 GB) files written to Parquet via PyArrow are corrupted Key: ARROW-2369 URL: https://issues.apache.org/jira/browse/ARROW-2369 Project: Apache Arrow

[jira] [Updated] (ARROW-2369) Large (>~20 GB) files written to Parquet via PyArrow are corrupted

2018-03-30 Thread Justin Tan (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tan updated ARROW-2369: -- Description: When writing large Parquet files (above 20 GB or so) from Pandas to Parquet via the comman

[jira] [Updated] (ARROW-2369) Large (>~20 GB) files written to Parquet via PyArrow are corrupted

2018-03-30 Thread Justin Tan (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tan updated ARROW-2369: -- Description: When writing large Parquet files (above 10 GB or so) from Pandas to Parquet via the comman

[jira] [Commented] (ARROW-2372) ArrowIOError: Invalid argument

2018-03-31 Thread Justin Tan (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421269#comment-16421269 ] Justin Tan commented on ARROW-2372: --- I'm experiencing the same issue with conversion of

[jira] [Commented] (ARROW-2369) Large (>~20 GB) files written to Parquet via PyArrow are corrupted

2018-04-09 Thread Justin Tan (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430571#comment-16430571 ] Justin Tan commented on ARROW-2369: --- Looks like the file is readable by early pyarrow ve