ahmedabu98 opened a new pull request, #41012: URL: https://github.com/apache/beam/pull/41012
Use RecordWriterManager to write through the HASH distribution mode. RecordWriterManager determines a destination file's partition from its rows instead of an incoming partition path. Using the parsed partition path directly resulted in some errors: - `year` partitions stored the calendar year instead of years since 1970 (2024 instead of 54). - Strings with spaces or special characters stayed URL-encoded (`a b/c` became `a+b%2Fc`). - A null string became the string `"null"`, so it shared a partition with real `"null"` values. - A null boolean became `false`. - Binary values became the bytes of their base64 text. - Nulls in numeric and date partitions, including `year`, `month`, `day` and `hour` partitions, crashed the write. - Identity partitions on timestamp or time columns crashed the write. Added tests to make sure these are covered against future changes. RecordWriterManager now numbers files per partition path, so a null and the string “null” no longer write the same file. Also fixed Timestamp.MICROS, which used to crash. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
