[
https://issues.apache.org/jira/browse/SPARK-30023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShivaKumar SS updated SPARK-30023:
----------------------------------
Labels: dataframe (was: )
> Spark partitionby saves as columnName={value} | Can it be only columnvalue
> --------------------------------------------------------------------------
>
> Key: SPARK-30023
> URL: https://issues.apache.org/jira/browse/SPARK-30023
> Project: Spark
> Issue Type: Question
> Components: Spark Core, SQL
> Affects Versions: 2.4.3
> Reporter: ShivaKumar SS
> Priority: Major
> Labels: dataframe
>
> I am using scala and spark.
> This is using Dataframe and in dataframe i have a columns by name "year"
> "month" and "date" and many other columns which are not relevant here.
>
> Code snippet.
>
> {{df.write.partitionBy("year", "month",
> "date").format("csv").option("header", "true").save(outPath)
> }}
>
> and my expectation is to save in a hierarchy folder structure.
>
> {{2016/11/15/file*.csv}}
>
> but the files are getting saved as
>
>
> {{year=2016/month=11/date=15/file*.csv}}
>
> {{Is there any way i can remove the column name from the directory structure
> and save only the column value here. ? }}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
