Sanford Ryza created SPARK-54358:
------------------------------------
Summary: Checkpoint dirs collide when streaming tables in
different schemas have same name
Key: SPARK-54358
URL: https://issues.apache.org/jira/browse/SPARK-54358
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Sanford Ryza
A streaming table is a table fed by a stream. Streaming tables have checkpoint
directories underneath their pipeline's storage root. These directories don't
currently take the table namespace into account, which means that two tables
with different schemas but the same name will be mapped to the same checkpoint
directory. This could be very bad and cause data loss.
We should update the per-streaming table checkpoint path to use the fully
qualified table path, instead of just its name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
