[
https://issues.apache.org/jira/browse/HUDI-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-8183:
----------------------------
Status: In Progress (was: Open)
> Record key value is null if the specified field does not exist
> --------------------------------------------------------------
>
> Key: HUDI-8183
> URL: https://issues.apache.org/jira/browse/HUDI-8183
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Assignee: Jonathan Vexler
> Priority: Blocker
> Fix For: 1.0.0
>
>
> When running TestCOWDataSourceStorage#testCopyOnWriteStorage with
> "true|org.apache.hudi.keygen.ComplexKeyGenerator|_row_key,nation.bytes" as
> the parameters, the test does not fail, but the record key is null in the
> target table, because "nation.bytes" field does not exist with Avro 1.11 (see
> HUDI-4240). The test will be fix by HUDI-4240, but if the field is not
> found, an error should be thrown instead of silently using "null" which
> causes data corruption.
> {code:java}
> +-------------------+----------------------+-------------------------------------------------------------------+----------------------+------------------------------------------------------------------------+------------------+------------------------------------+-------------------+-------------------+-------------+------------+----------+------------------+----------+--------------------+-------------------+-------------------------+--------------------+------+----------+--------------+---------+--------------------+---------+---------------------------+---------+-----------+
> |_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key
> |_hoodie_partition_path|_hoodie_file_name
>
> |_hoodie_is_deleted|_row_key |begin_lat
> |begin_lon
> |city_to_state|current_date|current_ts|distance_in_meters|driver |end_lat
> |end_lon |fare |height
> |nation|partition |partition_path|rider |seconds_since_epoch
> |timestamp|tip_history |trip_type|weight |
> +-------------------+----------------------+-------------------------------------------------------------------+----------------------+------------------------------------------------------------------------+------------------+------------------------------------+-------------------+-------------------+-------------+------------+----------+------------------+----------+--------------------+-------------------+-------------------------+--------------------+------+----------+--------------+---------+--------------------+---------+---------------------------+---------+-----------+
> |20240908111715094 |20240908111715094_0_0
> |_row_key:573b0bbe-466d-4d7d-9481-6c346c20b357,nation.bytes:__null__|2016/03/15
>
> |9419cd48-1862-4f87-beee-d61ed252cdf6-0_0-13-18_20240908111715094.parquet|false
>
> |573b0bbe-466d-4d7d-9481-6c346c20b357|0.21293608202267056|0.4616147611371665
> |{CA} |14 |1292455350|1756055707
> |driver-000|0.6341198556373024 |0.9730147837604404 |{13.665512257442803,
> USD}|[0, 0, 5, -75, -19] |Canada|2016/03/15|2016/03/15
> |rider-000|7785243976339578476 |0 |[{22.4026513706999, USD}] |UBERX
> |0.13111544 |
> |20240908111715094 |20240908111715094_0_1
> |_row_key:35018991-9801-4011-baaf-3e8e293729a1,nation.bytes:__null__|2016/03/15
>
> |9419cd48-1862-4f87-beee-d61ed252cdf6-0_0-13-18_20240908111715094.parquet|false
> |35018991-9801-4011-baaf-3e8e293729a1|0.8823883346258744
> |0.4740144304605969 |{CA} |13 |1154814627|976980522
> |driver-000|0.32209102499429076 |0.8129073772715241 |{13.056539131134503,
> USD}|[0, 0, 6, -120, 25] |Canada|2016/03/15|2016/03/15
> |rider-000|4492930791480814834 |0 |[{87.13256649606845, USD}] |BLACK
> |0.5812139 |
> |20240908111715094 |20240908111715094_0_2
> |_row_key:081658df-888b-40f4-bdb1-42bed194ead2,nation.bytes:__null__|2016/03/15
>
> |9419cd48-1862-4f87-beee-d61ed252cdf6-0_0-13-18_20240908111715094.parquet|false
> |081658df-888b-40f4-bdb1-42bed194ead2|0.637497895387057
> |0.7609280000096621 |{CA} |15 |1318756073|-1598001864
> |driver-000|0.7084929694473843 |0.7639075214983112 |{20.65661387074158, USD}
> |[0, 0, 11, -54, -96]|Canada|2016/03/15|2016/03/15
> |rider-000|-8045010737612317379|0 |[{85.62945468793872, USD}] |UBERX
> |0.09829235 |
> |20240908111715094 |20240908111715094_0_3
> |_row_key:f8a77def-b4cb-4282-9f02-c2ade771a3de,nation.bytes:__null__|2016/03/15
>
> |9419cd48-1862-4f87-beee-d61ed252cdf6-0_0-13-18_20240908111715094.parquet|false
> |f8a77def-b4cb-4282-9f02-c2ade771a3de|0.4231552109657305
> |0.7871730203087489 |{CA} |11 |980487792 |-2093868483
> |driver-000|0.5821179162472624 |0.17473910868905307|{40.37203787898523, USD}
> |[0, 0, 7, -10, 88] |Canada|2016/03/15|2016/03/15
> |rider-000|3154644657160133271 |0 |[{79.79239036016631, USD}] |BLACK
> |0.032471836|
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
