[
https://issues.apache.org/jira/browse/SPARK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622814#comment-16622814
]
Daniel Mateus Pires commented on SPARK-25480:
---------------------------------------------
* Didn't try accessing S3 without EMR
* EMRFS consistent view:Disabled
Release label:emr-5.13.0
Hadoop distribution:Amazon 2.8.3
Applications:Spark 2.3.0, Zeppelin 0.7.3, Hive 2.3.2
I'll try and reproduce without using EMR, and without using S3 and will update
the ticket
> Dynamic partitioning + saveAsTable with multiple partition columns create
> empty directory
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-25480
> URL: https://issues.apache.org/jira/browse/SPARK-25480
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Daniel Mateus Pires
> Priority: Minor
> Attachments: dynamic_partitioning.json
>
>
> We use .saveAsTable and dynamic partitioning as our only way to write data to
> S3 from Spark.
> When only 1 partition column is defined for a table, .saveAsTable behaves as
> expected:
> - with Overwrite mode it will create a table if it doesn't exist and write
> the data
> - with Append mode it will append to a given partition
> - with Overwrite mode if the table exists it will overwrite the partition
> If 2 partition columns are used however, the directory is created on S3 with
> the SUCCESS file, but no data is actually written
> our solution is to check if the table doesn't exist, and in that case, set
> the partitioning mode back to static before running saveAsTable:
> {code}
> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> df.write.mode("overwrite").partitionBy("year", "month").option("path",
> "s3://hbc-data-warehouse/integration/users_test").saveAsTable("users_test")
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
