Vlad Rozov created SPARK-50854:
----------------------------------
Summary: Usage of relative path in `DataStreamWriter` writes
parquet files in unexpected location
Key: SPARK-50854
URL: https://issues.apache.org/jira/browse/SPARK-50854
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.5.4
Reporter: Vlad Rozov
The following code writes {{_spark_metadata}} to a directory relative to the
spark driver, while parquet files are written to a directory relative to an
executor. Both should be written to a directory relative to the driver working
directory.
{code:java}
sparkSession.readStream()
.format("rate")
.load()
.writeStream()
.format("parquet")
.option("checkpointLocation", "checkpoints")
.trigger(AvailableNow())
.start("test.parquet")
.awaitTermination();
{code}
Please see https://lists.apache.org/thread/ffzwn1y2fgyjw0j09cv4np9z00wymxwv
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
