Re: Convert raw data files to Parquet format

2014-07-23 Thread buntu
That seems to be the issue, when I reduce the number of fields it works perfectly fine. Thanks again Michael.. that was super helpful!! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Convert-raw-data-files-to-Parquet-format-tp10526p10541.html Sent from

Re: Convert raw data files to Parquet format

2014-07-23 Thread Michael Armbrust
at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> >> Is this due to the amount of data (about 5M rows) being processed? I've >> set >> the SPARK_DRIVER_MEMORY to 8g. >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Convert-raw-data-files-to-Parquet-format-tp10526p10536.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >

Re: Convert raw data files to Parquet format

2014-07-23 Thread Michael Armbrust
til.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > Is this due to the amount of data (about 5M rows) being processed? I've set > the SPARK_DRIVER_MEMORY to 8g. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Convert-raw-data-files-to-Parquet-format-tp10526p10536.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Convert raw data files to Parquet format

2014-07-23 Thread buntu
java:745) Is this due to the amount of data (about 5M rows) being processed? I've set the SPARK_DRIVER_MEMORY to 8g. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Convert-raw-data-files-to-Parquet-format-tp10526p10536.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Convert raw data files to Parquet format

2014-07-23 Thread Michael Armbrust
onvert them to Parquet > format. I'm using standalone spark-shell. > > Thanks! > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Convert-raw-data-files-to-Parquet-format-tp10526.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Convert raw data files to Parquet format

2014-07-23 Thread buntu
t-raw-data-files-to-Parquet-format-tp10526.html Sent from the Apache Spark User List mailing list archive at Nabble.com.