nanjeshramesh opened a new pull request, #20556:
URL: https://github.com/apache/hudi/pull/20556

   ### Describe the issue this Pull Request addresses
   
   Closes #20504
   
   A `MERGE INTO` whose source has a `tinyint` or `smallint` column fails in 
`ExpressionPayload` with `ClassCastException: MutableInt cannot be cast to 
MutableByte` when the matched action is `UPDATE SET *`, even though the target 
column is `int`.
   
   ### Summary and Changelog
   
   The conditions and assignments of a `MERGE INTO` are evaluated against a row 
that `ExpressionPayload` deserializes from Avro. Hudi's schema has no 8 or 16 
bit integer type, so `tinyint` and `smallint` columns arrive in that row as 
`int`. `MergeIntoHoodieTableCommand` bound the expressions with the column's 
Spark type instead, so the generated code called `getByte` or `getShort` on an 
int slot.
   
   `bindReferences` now binds references of type `ByteType` or `ShortType` as 
`IntegerType` and casts them back to the original type, so the rest of the 
expression keeps its types. The change is made in both the Spark 3 and Spark 4 
copies of `MergeIntoHoodieTableCommand`.
   
   Added `Test MergeInto with tinyint and smallint source columns assigned to 
int columns` to `TestMergeIntoTable`. It uses `tinyint` and `smallint` source 
columns in a matched condition, a matched `UPDATE SET *` and a not matched 
`INSERT *`. It fails on master with the exception from the issue and passes 
with this change. The rest of `TestMergeIntoTable` and `TestMergeIntoTable2` 
still passes locally on Spark 3.5.
   
   ### Impact
   
   `MERGE INTO` statements with `tinyint` or `smallint` source columns no 
longer fail. Other types are bound as before.
   
   ### Risk Level
   
   low. Only references typed `ByteType` or `ShortType` are rewritten, and the 
cast back to the original type keeps the expression types unchanged. The Spark 
4 copy of the file has the same change and was not compiled locally.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): read tinyint a... nanjeshramesh (via GitHub) via commits

Reply via email to