[
https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201863#comment-16201863
]
Steve Loughran commented on SPARK-22240:
----------------------------------------
thanks. Now for a question which is probably obvious to others:
Is there a straightforward way to determine the number of partitions a file is
being broken into other than doing a job & counting the # of part- files? I
guess even if there is, counting the files is straightforward.
If it's multiline related, do you think this would apply to all filesystems, or
is it possible that s3a is making things worse?
Anyway, whatever happens here, we'll get a fix for s3a's getBlockLocations in
shortly
> S3 CSV number of partitions incorrectly computed
> ------------------------------------------------
>
> Key: SPARK-22240
> URL: https://issues.apache.org/jira/browse/SPARK-22240
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Environment: Running on EMR 5.8.0 with Hadoop 2.7.3 and Spark 2.2.0
> Reporter: Arthur Baudry
>
> Reading CSV out of S3 using S3A protocol does not compute the number of
> partitions correctly in Spark 2.2.0.
> With Spark 2.2.0 I get only partition when loading a 14GB file
> {code:java}
> scala> val input = spark.read.format("csv").option("header",
> "true").option("delimiter", "|").option("multiLine",
> "true").load("s3a://<s3_path>")
> input: org.apache.spark.sql.DataFrame = [PARTY_KEY: string, ROW_START_DATE:
> string ... 36 more fields]
> scala> input.rdd.getNumPartitions
> res2: Int = 1
> {code}
> While in Spark 2.0.2 I had:
> {code:java}
> scala> val input = spark.read.format("csv").option("header",
> "true").option("delimiter", "|").option("multiLine",
> "true").load("s3a://<s3_path>")
> input: org.apache.spark.sql.DataFrame = [PARTY_KEY: string, ROW_START_DATE:
> string ... 36 more fields]
> scala> input.rdd.getNumPartitions
> res2: Int = 115
> {code}
> This introduces obvious performance issues in Spark 2.2.0. Maybe there is a
> property that should be set to have the number of partitions computed
> correctly.
> I'm aware that the .option("multiline","true") is not supported in Spark
> 2.0.2, it's not relevant here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
