[
https://issues.apache.org/jira/browse/ARROW-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111787#comment-16111787
]
Keith Curtis edited comment on ARROW-1311 at 8/2/17 10:00 PM:
--------------------------------------------------------------
I re-ran my code, and have a revised function, where I added a line to update
the column, which seems to matter.
def to_parquet(output_file, csv_file):
df = pd.read_csv(csv_file)
df['gecco_variant'] = [ v.lstrip('0') for v in df['gecco_variant']]
table = pyarrow.Table.from_pandas(df)
pq.write_table(table, output_file)
When Python seemed hung (after 3 minutes with no progress), I captured a stack
trace with gdb, and attached the file
I'm running on Ubuntu 14.04.3. I installed into a conda virtual environment
using pip.
was (Author: k94):
I re-ran my code, and and have a revised function
def to_parquet(output_file, csv_file):
df = pd.read_csv(csv_file)
df['gecco_variant'] = [ v.lstrip('0') for v in df['gecco_variant']]
table = pyarrow.Table.from_pandas(df)
pq.write_table(table, output_file)
When Python seemed hung (after 3 minutes with no progress), I captured a stack
trace with gdb, and attached the file
I'm running on Ubuntu 14.04.3. I installed into a conda virtual environment
using pip.
> python hangs after write a few parquet tables
> ---------------------------------------------
>
> Key: ARROW-1311
> URL: https://issues.apache.org/jira/browse/ARROW-1311
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.5.0
> Environment: Python 3.5.2, pyarrow 0.5.0
> Reporter: Keith Curtis
> Assignee: Wes McKinney
> Fix For: 0.6.0
>
> Attachments: backtrace.txt
>
>
> I had a program to read some csv files (a few million rows each, 9 columns),
> and converted with:
> ```python
> import os
> import pandas as pd
> import pyarrow.parquet as pq
> import pyarrow
> def to_parquet(output_file, csv_file):
> df = pd.read_csv(csv_file)
> table = pyarrow.Table.from_pandas(df)
> pq.write_table(table, output_file)
> ```
> The first csv file would always complete, but python would hang on the second
> or third file, and sometimes on a much later file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
