kz930 commented on PR #8341: URL: https://github.com/apache/texera/pull/8341#issuecomment-5735789839
Thanks, all five are fixed in `32173ef6f`. Booleans go through one shared parser matching `parseField`, tested against the executor for `1` and for invalid text. Parallel CSV no longer slices, since its executor leaves limit and offset as TODOs. Each source now declares the path it reads and the translator names them across the plan, numbering the second of two files whose paths end alike. Extraction walks the zip entries, skipping the ones macOS adds and taking the entry's own name. Parallel CSV and Old CSV keep a literal NA and take their column names from the schema. Nullable longs stay exact in the main CSV, JSON Lines and Arrow. Two fixes are deliberately not the literal "same as the main CSV source", because the three readers disagree about a blank field. Old CSV names no missing value: scala-csv hands a blank back as the empty string, so `na_values` would null a value it keeps. Parallel CSV asks for the large-integer column as text rather than as a nullable integer, because its schema types that column text while its executor nulls the blank. `read_json` was not fixable with a dtype, since it rounds the number before any dtype applies. That source rebuilds those columns from an exact parse of the same lines. Two engine defects are filed rather than worked around: #8596 and #8598. The export follows the panel on the first, so it diverges from the engine until that one is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
