AngersZhuuuu commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733453929
> Let's revisit the entire process: > > 1. the parser creates partition spec which is `Map[String, String]` > 2. for v1 commands, we pass `Map[String, String]` to hive APIs. > 3. for v2 commands, we need to convert string type partition values back to the actual type, construct an `InternalRow` and pass it to v2 APIs. > > It has several problems: > > 1. unnecessary string <-> actual type roundtrip in v2 commands. > 2. string can't represent all values. e.g. how to distinguish from null and "null"? > > My proposal is: > > 1. parser creates partition spec as `Map[String, Literal]` > 2. v1 commands can simply use `Cast` operator to turn `Literal` to string. > 3. v2 commands can simply evaluate `Literal` to construct `InternalRow` > > cc @rdblue @brkyvz I have through about just return Map[String, Literal] in parser level too but since a big API change I didn't do that in this pr. Return a literal have stronger and more explicit expression meaning. If follow this proposal, I can work on this ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
