jayzhan211 commented on issue #11513: URL: https://github.com/apache/datafusion/issues/11513#issuecomment-2247366030
> > Instead of LogicalPhysicalType + LogicalPhysicalSchema to keep track of the relationship between the logical and physical type, could we get the unidirectional mapping of logical type from physical type? > > Oh now I understand. It makes sense to do it this way but it would lose the ability of potentially adding new [arrow-compatible physical encodings](https://arrow.apache.org/docs/format/Columnar.html#extension-types) (e.g. logical `Utf8` with physical `List(UInt8)`) Great, a step close to it. I think we could consider this kind of case as `user defined type` where there is a UserDefinedType that built on top of our datafusion's primitive type, in this case `Utf8`. But we actually have `List(Uint8)` inside arrow::Schema, so we can decode the type with List(UInt8). Therefore, we have `UserDefinedType(Utf8)` in the logical planning step, and `List(Uint8)` in the physical planning step -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
