Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21308
@tigerquoll, I'm talking about the DataSourceV2 API in general. I'm not
sure if I think there is value in exposing partitions, but I'd be happy to hear
why you think they are valuable and think through how it would fix with the
existing API.
I think that partitions that aren't hidden make tables much harder for
users to work with, which is why Iceberg hides partitioning and automatically
translates from row filters to partition filters. For Kudu, maybe it is
different. Could you write up the use case with a bit more context about what
empty partitions are used for, and send it to the dev list?
If we think that the v2 API should expose a partition concept, then that
would definitely include a way to add or drop partitions.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
