HyukjinKwon commented on PR #57899: URL: https://github.com/apache/spark/pull/57899#issuecomment-5247722807
Thanks, @dongjoon-hyun — great catch, this was a real runtime bug. Fixed in 2a8cde439730. **Root cause**: `empty_type` was updated only inside the `if len(chunk):` guard, so a zero-length chunk (from an input batch of only empty/null arrays) left it at the UTC-typed default. The rows emitted for that batch used the UTC type, then a later real chunk from the pandas flavor (session-timezone-typed) made the next output batch a different schema, and `ArrowStreamSerializer.dump_stream` rejected it with `ArrowInvalid: Tried to write record batch with different schema`. **Fix**: as you suggested, take `empty_type` from every chunk (before the non-empty guard) — a zero-length chunk still carries the flavor's element type, so all emitted output batches stay on one schema. **Test**: added your `test_scalar_pandas_iter_udf_timestamp_after_empty_batch` reproduction. I adapted the assertion to compare against the equivalent non-iterator pandas UDF instead of an absolute datetime literal, since under `America/Los_Angeles` the driver collects the tz-naive midnight back as a UTC-shifted wall-clock (`17:00`), which made the hard-coded `datetime(2020, 1, 2)` mismatch; comparing the two UDF forms goes through identical driver-collection semantics and still proves the schema fix. I verified it reproduces the `ArrowInvalid` on the pre-fix revision and passes after. Full HOF UDF suite green (53), and `ruff format`/`ruff check` + mypy clean on `worker.py`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
