ammarchalifah commented on PR #58013: URL: https://github.com/apache/spark/pull/58013#issuecomment-6131503770
Thank you for working on this, I really find the feature important and it's motivating to see the progress. I want to clarify one of your last comments > Added Seq(1) over (int, string) partition keys. One detail: it doesn't fail in the hash. InterpretedHashFunction.hash matches on the value rather than the declared type, so the int under a StringType just hashes as an int and gets through. It fails in the ordering instead, once the raw key's hash collides with a projected one: java.lang.ClassCastException: class java.lang.Integer cannot be cast to class org.apache.spark.unsafe.types.UTF8String at BaseGenericInternalRow.getUTF8String, from InternalRowComparableWrapper.equals. Does this mean even with this fix, a MERGE on a table partitioned on `string_column_1, bucket(N, string_column_2)`, with join key only on `string_column_2` would still face an ordering issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
