JingsongLi commented on PR #8282: URL: https://github.com/apache/paimon/pull/8282#issuecomment-4747256058
Thanks for adding Parquet vector support. One correctness concern: VectorType is fixed-size, but this implementation encodes it as a Parquet LIST and currently accepts whatever list length is present. On the write path, ParquetRowDataWriter writes row.getVector(...).size() without checking it against VectorType.getLength(); on the read path, CastedVectorColumnVector returns a ColumnarVec using the Parquet list length, not the declared vector length. That can let malformed vectors be written/read and then fail later in Spark/DataConverter or vector indexing with less context. Could we validate the vector length at the Parquet boundary, and ideally add a test for mismatched length? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
