vinishjail97 commented on PR #828: URL: https://github.com/apache/incubator-xtable/pull/828#issuecomment-4772180624
@the-other-tim-brown The three partition columns are all generated from one source (`event_time`), so a row with a null `event_time` makes year/month/day **all** null at once — not one-of-three. It's just a nullable-timestamp scenario (optional / late-arriving / backfilled rows; generated columns are nullable unless constrained). On write, Delta puts that file under `…=__HIVE_DEFAULT_PARTITION__/…` and parses it back into the AddFile `partitionValues` as genuine nulls — `DelayedCommitProtocol.parsePartitions` ([v2.4.0 L121-146](https://github.com/delta-io/delta/blob/v2.4.0/core/src/main/scala/org/apache/spark/sql/delta/files/DelayedCommitProtocol.scala#L121-L146)) casts the NULL literal produced by `PartitionUtils` ([v2.4.0 L586-589](https://github.com/delta-io/delta/blob/v2.4.0/core/src/main/scala/org/apache/spark/sql/delta/util/PartitionUtils.scala#L586-L589)) to a null string. On read both Delta (kernel `literalForPartitionValue`, [v4.0.0 L479-482](https://github.com/delta-io/delta/blob/v4.0.0/kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/PartitionUtils.java#L479-L482)) and xtable take partition values from that map, where they are nulls — the `__HIVE_DEFAULT_PARTITION__` sentinel only ever lives in the physical path. So with all three components null, the old composite branch did `Collectors.joining("-")` over `[null, null, null]`, which yields the literal string `"null-null-null"` (StringJoiner appends `"null"` for null elements — no NPE). `convertFromDeltaPartitionValue` then skips its null guard and, since this is a `DAY` transform, runs `formatter.parse("null-null-null")`, which throws `org.apache.xtable.model.exception.ParseException: Unable to parse partition value`. The fix returns null for the composite when any component is null, matching Delta's own read semantics. This was never caught before because no existing partitioned test exercised a null partition value: `testVariousOperations` uses a single-field `level:VALUE` partition (never the multi-field join, never null), and the only generated partition (`YEAR(birthDate)`) is a single column. The new test is the first to hit the composite branch with a null source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
