[
https://issues.apache.org/jira/browse/SPARK-61826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146476#comment-18146476
]
Andreas Neumann commented on SPARK-61826:
-----------------------------------------
The description is mot accurate. The catalog setting in the generated pipeline
spec has no influence in the catalog implementation. It only names the default
catalog in the metastore.
SDP was developed on top of Spark Connect, as a true client-server app, which
means that the Spark Connect server is configured with the catalog
implementation, and the `spark-pipelines` command must provide a `--remote
sc://<server:port>` (If the server uns locally, then this is typically
`sc://localhost:15002`.
In addition, the Spark Connect server itself must use a catalog implementation
that persists metadata. Otherwise, every restart of the server will lose all
metadata, and the next run of the pipeline wiill again fail. Since SDP is built
on top of DSv2, it also requires that the catalog implements row-level updates.
This is currently only true for Iceberg, so we should recommend that to use
Hive with Iceberg as the default format.
> Projects generated by spark-pipelines init lose their tables after the run
> and fail the second run with LOCATION_ALREADY_EXISTS
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61826
> URL: https://issues.apache.org/jira/browse/SPARK-61826
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.3.0
> Reporter: Sudhanva Huruli
> Priority: Major
> Labels: pull-request-available
>
> spark-pipelines init writes a spec without a catalog setting, so a run uses
> the in-memory session catalog. When the run ends the tables are forgotten,
> but their folders stay under spark-warehouse/, so the second spark-pipelines
> run fails with LOCATION_ALREADY_EXISTS and the tables cannot be queried from
> spark-sql or pyspark after the run. The generated spark-pipeline.yml should
> set spark.sql.catalogImplementation: hive so metadata persists in a local
> metastore_db/.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]