cloud-fan commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733429639
Let's revisit the entire process: 1. the parser creates partition spec which is `Map[String, String]` 2. for v1 commands, we pass `Map[String, String]` to hive APIs. 3. for v2 commands, we need to convert string type partition values back to the actual type, construct an `InternalRow` and pass it to v2 APIs. It has several problems: 1. unnecessary string <-> actual type roundtrip in v2 commands. 2. string can't represent all values. e.g. how to distinguish from null and "null"? My proposal is: 1. parser creates partition spec as `Map[String, Literal]` 2. v1 commands can simply use `Cast` operator to turn `Literal` to string. 3. v2 commands can simply evaluate `Literal` to construct `InternalRow` cc @rdblue @brkyvz ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
