rangareddy commented on issue #19506: URL: https://github.com/apache/hudi/issues/19506#issuecomment-5349117339
Recording a side effect of #19463 here, as agreed in [that review thread](https://github.com/apache/hudi/pull/19463#discussion_r3765218370), so whoever picks this up does not assume the loud failure is still there. The count check in `orderFields` was incidentally catching *some* of this divergence. #19463 makes `cleanProjectionColumnIds` drop every blank read-column id from the conf, so inputs that used to trip the count check now resolve to the mis-paired order instead. Worked example, every step a real Hive/Hudi call on one JobConf: ``` Hudi addProjectionField x2 ids "2,0" names "_hoodie_record_key,_hoodie_commit_time" Hive appendReadColumns([7],["g"]) ids "7,2,0" names "_hoodie_record_key,_hoodie_commit_time,g" ^ ids prepend, names append -- the divergence in this issue Hive appendReadColumns([],[]) ids ",7,2,0" Hive appendReadColumns([],[]) ids ",,7,2,0" cleanProjectionColumnIds before #19463: ids ",7,2,0" (strips one leading comma only) after #19463: ids "7,2,0" (drops every blank) orderFields("_hoodie_record_key,_hoodie_commit_time,g", <ids>) before: throws #fieldNames: 3, #fieldPositions: 4 after: returns [g, _hoodie_commit_time, _hoodie_record_key] correct pairing is [_hoodie_commit_time, _hoodie_record_key, g] ``` So the blank no longer reaches `orderFields` at all and the count check is bypassed one step earlier than it was. The mis-pairing itself is unchanged by #19463; it is the positional pairing of two independently de-duplicated lists described above. Downstream appears to resolve by name, which is likely why this is masked today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
