kita-renji commented on issue #51499: URL: https://github.com/apache/arrow/issues/51499#issuecomment-5853851421
I can reproduce this on Linux with pyarrow 23.0.1 and 25.0.1, and the code path is unchanged on current main (51955ec). It isn't specific to `%s`. Any specifier the vendored date library doesn't know is copied to the output as-is, for timestamp, date32 and time64: `%s`, `%k`, `%l`, `%P`, flags like `%-d`, most `%E?`/`%O?` combinations, and undefined ones like `%i` or a trailing `%`. `Strftime::Make` only checks for `%c` and `%z`/`%Z` (https://github.com/apache/arrow/blob/51955ecd2c0bb94b55c2130171386f8568fc2dbe/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc#L1207-L1225), and everything else goes to date.h's `to_stream`, whose `default:` case writes unknown specifiers back out (https://github.com/apache/arrow/blob/51955ecd2c0bb94b55c2130171386f8568fc2dbe/cpp/src/arrow/vendored/datetime/date.h#L6104-L6116). On Windows the `std::format` backend should throw for these instead, so the same format likely errors there and silently echoes elsewhere. Those checks are also plain substring searches, so `%Ez` on naive input prints `+00:00` and `%%z` wrongly raises "Timezone not present". I'd suggest having `Make` tokenize the format and return `Invalid` for any specifier outside the supported set, with the `%c` and `%z`/`%Z` checks in the same pass. That covers every specifier and both substring cases. The catch is that formats which currently "work" by echoing would start raising. Implementing `%s` could be a follow-up, since its semantics need a decision: glibc and Python use the local timezone, which is why the expected value in the report (1609500645) is an hour off the UTC epoch (1609504245). Until then, `pc.cast(pc.cast(arr, pa.timestamp("s")), pa.int64())` gives epoch seconds. (`%S` printing `45.000000` is documented: the array is `timestamp[us]`.) I have the validation change and a C++ test working locally. Happy to open a PR with a Python test if this direction sounds right. Investigated with help from Claude Code; I verified the repro and results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
