wilhelm-ubm opened a new issue, #57897:
URL: https://github.com/apache/spark/issues/57897
pyspark=4.1.2
iceberg=1.11.0
Given the code:
```python
import pyspark.pipelines as dp
spark = ...
@dp.materialized_view
def table1():
return spark.table("existing_table")
@dp.materialized_view
def table2():
return spark.table("table1")
```
When running `spark-pipelines run` when the MV's do not already exist it
successfully determines that table2 needs to be created after table1
```
<timestamp> Flow <namespace>.table1 is QUEUED
<timestamp> Flow <namespace>.table2 is QUEUED
<timestamp> Flow <namespace>.table1 is PLANNING
<timestamp> Flow <namespace>.table1 is STARTING
<timestamp> Flow <namespace>.table1 is RUNNING
<timestamp> Flow <namespace>.table1 has COMPLETED
<timestamp> Flow <namespace>.table2 is PLANNING
...
```
However, this does not happen when running the spark pipeline again when the
mv's/tables already exist:
```
<timestamp> Flow <namespace>.table1 is QUEUED
<timestamp> Flow <namespace>.table2 is QUEUED
<timestamp> Flow <namespace>.table2 is PLANNING
<timestamp> Flow <namespace>.table2 is STARTING
...
<timestamp> Flow <namespace>.table1 is PLANNING
...
```
We would assume that the initial behaviour that takes into consideration the
dependencies would be respected when you run the pipelines multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
