[
https://issues.apache.org/jira/browse/HUDI-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuo Cheng updated HUDI-9603:
-----------------------------
Fix Version/s: 1.1.1
(was: 1.1.0)
> Flink 1.18 array values incorrect in fg reader test
> ---------------------------------------------------
>
> Key: HUDI-9603
> URL: https://issues.apache.org/jira/browse/HUDI-9603
> Project: Apache Hudi
> Issue Type: Bug
> Components: flink, reader-core
> Affects Versions: 1.0.2
> Reporter: Jonathan Vexler
> Priority: Blocker
> Fix For: 1.1.1
>
>
> {code:java}
> <{"timestamp": 0, "_row_key": "30f916be-06d6-4ef3-a9fb-9dddc40e8abd",
> "partition_path": "any_partition", "trip_type": "BLACK", "rider":
> "rider-001", "driver": "driver-001", "begin_lat": 0.7585664392201925,
> "begin_lon": 0.37789349980839115, "end_lat": 0.7757098794857531, "end_lon":
> 0.1730947142293684, "distance_in_meters": 1356260410, "seconds_since_epoch":
> -9002173267385180332, "weight": 0.9950906, "nation": "Canada",
> "current_date": 13, "current_ts": 1182257224, "height": [0, 0, 11, 121, -97],
> "city_to_state": {"LA": "CA"}, "fare": {"amount": 89.22144149794701,
> "currency": "USD"}, "tip_history": [{"amount": 78.67847272460095, "currency":
> "USD"}], "customField0": 1299566198, "customField1": 5724776041695989483,
> "customField2": 0.7481167, "customField3": 0.2553663812411848,
> "customField4": "25db73af-fdf9-4e04-b47d-ce44882bf362", "customField5":
> "877c2613-f2a1-4afd-af4a-863814dec38b", "_hoodie_is_deleted": false}>
> <{"timestamp": 0, "_row_key": "30f916be-06d6-4ef3-a9fb-9dddc40e8abd",
> "partition_path": "any_partition", "trip_type": "BLACK", "rider":
> "rider-001", "driver": "driver-001", "begin_lat": 0.7585664392201925,
> "begin_lon": 0.37789349980839115, "end_lat": 0.7757098794857531, "end_lon":
> 0.1730947142293684, "distance_in_meters": 1356260410, "seconds_since_epoch":
> -9002173267385180332, "weight": 0.9950906, "nation": "Canada",
> "current_date": 13, "current_ts": 1182257224, "height": [0, 0, 11, 121, -97],
> "city_to_state": {"LA": "CA"}, "fare": {"amount": 89.22144149794701,
> "currency": "USD"}, "tip_history": [{"amount": 4.5791476348245315,
> "currency": "USD"}], "customField0": 1299566198, "customField1":
> 5724776041695989483, "customField2": 0.7481167, "customField3":
> 0.2553663812411848, "customField4": "25db73af-fdf9-4e04-b47d-ce44882bf362",
> "customField5": "877c2613-f2a1-4afd-af4a-863814dec38b", "_hoodie_is_deleted":
> false}>
> {code}
> every field is the same except for the tip_history. This only happens on
> Flink 1.18 and not 1.20. I added a config to
> HoodieTestDataGenerator.SchemaEvolutionConfigs that can be enabled to expose
> this issue. I have marked it as a TODO in the code
> I read the data in spark to ensure it wasn't an issue with the writer:
> {code:java}
> scala>
> spark.read.format("hudi").load("/var/folders/d0/l7mfhzl1661byhh3mbyg5fv00000gn/T/junit-5387917324981749197/").select("_row_key","tip_history").show(100,false)
> 25/07/18 14:12:07 WARN DFSPropertiesConfiguration: Properties file
> file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
> 25/07/18 14:12:07 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR,
> please set it as the dir of hudi-defaults.conf
> # WARNING: Unable to attach Serviceability Agent. Unable to attach even with
> module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException:
> Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException:
> Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException:
> Sense failed.]
> +------------------------------------+---------------------------+
>
> |_row_key |tip_history |
> +------------------------------------+---------------------------+
> |30f916be-06d6-4ef3-a9fb-9dddc40e8abd|[{78.67847272460095, USD}] |
> |d0efa8ad-5f8e-4ee2-ace7-963585aa2b7f|[{94.05111350973606, USD}] |
> |f0dd358d-de15-4c67-9e3b-f006263d9286|[{30.4615021527871, USD}] |
> |e6db310d-71ca-4147-9f4f-cbf7a2675016|[{3.853885568405202, USD}] |
> |27819509-b6ac-45ee-8a7a-a031b2ef9a05|[{4.5791476348245315, USD}]|
> +------------------------------------+---------------------------+ {code}
> The tip history matches the expected 78.67847272460095 for record
> 30f916be-06d6-4ef3-a9fb-9dddc40e8abd.
> Additionally, we can see that the value that is showing up in the record
> exactly matches the value in the last record
> `27819509-b6ac-45ee-8a7a-a031b2ef9a05`
>
> I went back and looked at all the expected records and they all have that
> same value in their tip history
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
