mbutrovich commented on issue #1789: URL: https://github.com/apache/datafusion-comet/issues/1789#issuecomment-2922749343
The investigation seems to be diverging from `map_values` and seems more focused on map value nullability, and in particular struct field nullability and how that's encoded and read back in Parquet. If that's the case, I suggest starting from simpler questions without concerning maps: - Can a _struct_ that is non-nullable have nullable fields (I believe yes)? How is that encoded in Parquet by Spark, and how does DF treat that? - Can a _struct_ that is nullable have non-nullable fields (I believe yes)? How is that encoded in Parquet by Spark, and how does DF treat that? Once we know that I suggest we go back to the map issue with structs as keys and values and try to make more sense of what's going on there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
