On 11/6/15 10:53 PM, Rok Roskar wrote:
yes I was expecting that too because of all the metadata generation
and compression. But I have not seen performance this bad for other
parquet files I’ve written and was wondering if there could be
something obvious (and wrong) to do with how I’ve specified the schema
etc. It’s a very simple schema consisting of a StructType with a few
StructField floats and a string. I’m using all the spark defaults for
io compression.
I'll see what I can do about running a profiler -- can you point me to
a resource/example?
This link is probably helpful:
https://cwiki.apache.org/confluence/display/SPARK/Profiling+Spark+Applications+Using+YourKit
Thanks,
Rok
ps: my post on the mailing list is still listed as not accepted by the
mailing list:
http://apache-spark-user-list.1001560.n3.nabble.com/very-slow-parquet-file-write-td25295.html
-- none of your responses are there either. I am definitely subscribed
to the list though (I get daily digests). Any clue how to fix it?
Sorry, no idea :-/
On Nov 6, 2015, at 9:26 AM, Cheng Lian <lian.cs....@gmail.com
<mailto:lian.cs....@gmail.com>> wrote:
I'd expect writing Parquet files slower than writing JSON files since
Parquet involves more complicated encoders, but maybe not that slow.
Would you mind to try to profile one Spark executor using tools like
YJP to see what's the hotspot?
Cheng
On 11/6/15 7:34 AM, rok wrote:
Apologies if this appears a second time!
I'm writing a ~100 Gb pyspark DataFrame with a few hundred
partitions into a
parquet file on HDFS. I've got a few hundred nodes in the cluster,
so for
the size of file this is way over-provisioned (I've tried it with fewer
partitions and fewer nodes, no obvious effect). I was expecting the
dump to
disk to be very fast -- the DataFrame is cached in memory and
contains just
14 columns (13 are floats and one is a string). When I write it out
in json
format, this is indeed reasonably fast (though it still takes a few
minutes,
which is longer than I would expect).
However, when I try to write a parquet file it takes way longer --
the first
set of tasks finishes in a few minutes, but the subsequent tasks
take more
than twice as long or longer. In the end it takes over half an hour
to write
the file. I've looked at the disk I/O and cpu usage on the compute
nodes and
it looks like the processors are fully loaded while the disk I/O is
essentially zero for long periods of time. I don't see any obvious
garbage
collection issues and there are no problems with memory.
Any ideas on how to debug/fix this?
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/very-slow-parquet-file-write-tp25295.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail: user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
