wilhelmagren opened a new issue, #57908:
URL: https://github.com/apache/spark/issues/57908
spark=4.1.3
iceberg=1.11.0
### Problem
SDP currently treats every entry in `partition_cols` as an identity
partition transform.
for example
```python
@dp.materialized_view(partition_cols=["months(event_time)"])
def events():
...
```
causes `months(event_time)` to be interpreted as a column name instead of a
partition transform.
Behaviour comes from `DatasetManager.materializeTable` where all partition
cols are converted using identity
```scala
table.partitionCols.toSeq.flatten.map(Expressions.identity)
```
and as a result using the above partition col it just becomes
`identity('months(event_time)')`.
### Expected behaviour
SDP should support the same partition transforms that Spark SQL already
supports
Bare col-names (`partition_col=["event_date"]`) should continue producing
`identity(event_date)`
but partition transforms should be respected, and combining partitions like
```python
partition_cols=["country", "bucket(16, user_id)"]
```
should produce its corresponding V2 partition transforms.
### Existing SQL support
Catalyst parses partition transform syntax for SQL `PARTITIONED BY` and
creates corresponding V2 `Transform`
SDP should just reuse the existing Catalyst parsing logic and not
short-circuit everything to identity transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
