HyukjinKwon commented on pull request #31284: URL: https://github.com/apache/spark/pull/31284#issuecomment-768630015
@razajafri, do you mind clarifying PR description? For exmaple, I thought you meant writing out to files or somewhere by: > Spark should read it as a long but write it as an int by downcasting it and calling the appropriate method to set the integer value The change itself looks making sense but my question is how parquet-mr handles this case. Vectorized readers are supposed to be contribute back to Parquet side so it would be great if how they handle and we match it. It's more because it looks a bit odds to me that we should manipulate the column descriptor. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
