ShivaKumar SS created SPARK-30023:
-------------------------------------
Summary: Spark partitionby saves as columnName={value} | Can it be
only columnvalue
Key: SPARK-30023
URL: https://issues.apache.org/jira/browse/SPARK-30023
Project: Spark
Issue Type: Question
Components: Spark Core, SQL
Affects Versions: 2.4.3
Reporter: ShivaKumar SS
I am using scala and spark.
This is using Dataframe and in dataframe i have a columns by name "year"
"month" and "date" and many other columns which are not relevant here.
Code snippet.
{{df.write.partitionBy("year", "month", "day").format("csv").option("header",
"true").save(outPath)
}}
and my expectation is to save in a hierarchy folder structure.
{{2016/11/15/file.csv}}
but the files are getting saved as
{{year=2016/month=11/day=15/file.csv}}
{{}}
{{Is there any way i can remove the column name from the directory structure
and save only the column value here. ? }}
{{}}
{{}}
{{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
