ayushi-agarwal opened a new issue, #8663:
URL: https://github.com/apache/incubator-gluten/issues/8663
### Backend
VL (Velox)
### Bug description
val data = Seq(
("b1", 1, "a1"),
("b2", 2, "a2"),
("b1", 3, "a1"),
("b2", 4, "a2")
)
// Define schema explicitly
val schema = StructType(Seq(
StructField("b", StringType, nullable = false),
StructField("c", IntegerType, nullable = false),
StructField("a", StringType, nullable = false),
))
val rdd = spark.sparkContext.parallelize(data).map {
case (b, c, a) => Row(b, c, a)
}
val df = spark.createDataFrame(rdd, schema)
// Write DataFrame as Parquet with partitioning
df.write
.format("parquet")
.partitionBy("a", "b") // Partition by columns a and b
.mode("overwrite") // Overwrite if output exists
.save("file:///tmp/partitioned_output")
This creates directory structure as
b=b1/a=a1
b=b2/a=a2
instead of
a=a1/b=b1
a=a2/b=b2
@zhouyuan @JkSelf
### Spark version
None
### Spark configurations
spark - 3.5.1
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
