kz930 opened a new issue, #8511: URL: https://github.com/apache/texera/issues/8511
### Task Summary Texera reads CSV, JSONL, Arrow and plain text off disk, but not Parquet, which is the format most tables in a data-science workflow are already stored in. A user holding one has to convert it first, and a conversion to CSV loses the types the file was written with: the column that was an INTEGER comes back as text for the schema to guess at again. A scan source that reads Parquet closes that. It is the same shape as the CSV and Arrow sources, a `ScanSourceOpDesc` declaring `fileTypeName = "Parquet"`, and it reads its schema off the file's own footer rather than inferring one, so the types survive. `pyarrow` is already a declared dependency, and Arrow's source shows how a binary format is read. The parity harness covers it the moment the format is registered: `SourceCategoryRunner` maps a declared `fileTypeName` to an encoder that writes a fixture in it, and Parquet needs one entry added to `encoderByFileType`. This is the first operator to exercise that branch, the four already there being covered by an encoder that existed. Not part of #8325: nothing here makes a workflow exportable. It does ship standalone code, so it comes after the trait #8327 introduces, and its fixture needs the harness that issue builds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
