voonhous opened a new pull request, #20542: URL: https://github.com/apache/hudi/pull/20542
### Describe the issue this Pull Request addresses Fixes #20538 Stacked on #20541 (itself on #20540): review only the last commit, `fix(spark): pin expression index to its time zone`. An expression index on a zone dependent function (`hour`, `date_format`, or the cast Spark adds to parse a string) stored values in the writer's session time zone, and recorded no zone. A query in another zone pruned files holding matching rows, and writers in different zones mixed zones within one index. ### Summary and Changelog - `CREATE INDEX` records the session time zone as the index option `timeZone`, unless the user sets it. - Every write computes the index values in that zone (`ExpressionIndexTimeZone.inTimeZone`), whatever the writer's session zone. - `IndexedExpression.matches`: a zone dependent term reads the index only if it is evaluated in the index's zone (`UTC`, `Etc/UTC` and `+00:00` are the same zone). - An index with no recorded zone is not read for zone dependent terms. Others (`upper`, `length`, ...) are unaffected. ### Impact Fixes wrong results for queries in a zone other than the index's. Existing expression indexes on zone dependent functions stop pruning until they are dropped and re-created, which records the zone. Indexes created through write configs (`hoodie.metadata.index.expression.options`) record a zone only if `timeZone` is set there. ### Risk Level low. See details. <details> <summary>Tests and open point</summary> - `TestExpressionIndex#Test Expression Index Is Read Only In The Time Zone It Was Computed In` (the #20538 repro): index created in UTC, a row written from Los Angeles, then queried in UTC (prunes, the Los Angeles write is found by its UTC hour) and in Los Angeles (`hour(ts) = 18`, `hour(ts) = 2`, `date_format(...) = '2023-09-18'` return the matching rows; the index is not read). - `TestDataSkippingStages#testIndexedExpressionMatchesOnlyInItsTimeZone`: zone equivalence, the cast's zone, an index with no zone. - Spark 3.5: `TestExpressionIndex` (28), `TestDataSkippingStages` (28), `TestDataSkippingGoldenFilters` pass. - With the write-side pin removed, the new test fails at `hour(ts) = 12`: the row written from Los Angeles is stored under its Los Angeles hour and pruned. Open point: a Hudi writer from before this change ignores `timeZone` and computes in its own session zone, so it can still mix zones into an index that records one. Gating that needs an index or table version bump; not done here. </details> ### Documentation Update The expression index docs should list the `timeZone` option and the re-create step for existing indexes. Website follow-up. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
