[
https://issues.apache.org/jira/browse/HUDI-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522919#comment-17522919
]
sivabalan narayanan commented on HUDI-3749:
-------------------------------------------
regular hive sync worked out of the box.
{code:java}
df.write.format("hudi").
option(PRECOMBINE_FIELD_OPT_KEY, "tpep_dropoff_datetime").
option(RECORDKEY_FIELD_OPT_KEY, "tpep_pickup_datetime").
option(PARTITIONPATH_FIELD_OPT_KEY, "date_col").
option(TABLE_NAME, "hudi_tbl1").
option("hoodie.embed.timeline.server","false").
option("hoodie.datasource.hive_sync.enable","true").
option("hoodie.datasource.hive_sync.database","default").
option("hoodie.datasource.hive_sync.table","test_tbl3").
option("hoodie.datasource.hive_sync.mode","hms").
option("hoodie.datasource.hive_sync.partition_fields","_hoodie_partition_path").
mode(Overwrite).
save(basePath)
{code}
via beeline:
{code:java}
select * from test_tbl3 limit 5;{code}
{code:java}
+--------------------------------+---------------------------------+-------------------------------+----------------------------------------------------+---------------------+---------------------------------+----------------------------------+----------------------------+--------------------------+-----------------------+-------------------------------+-------------------------+-------------------------+-------------------------+------------------------+------------------+--------------------+-----------------------+-------------------------+----------------------------------+-------------------------+---------------------------------+---------------------+-----------------------------------+
| test_tbl3._hoodie_commit_time | test_tbl3._hoodie_commit_seqno |
test_tbl3._hoodie_record_key | test_tbl3._hoodie_file_name
| test_tbl3.vendorid | test_tbl3.tpep_pickup_datetime |
test_tbl3.tpep_dropoff_datetime | test_tbl3.passenger_count |
test_tbl3.trip_distance | test_tbl3.ratecodeid | test_tbl3.store_and_fwd_flag
| test_tbl3.pulocationid | test_tbl3.dolocationid | test_tbl3.payment_type
| test_tbl3.fare_amount | test_tbl3.extra | test_tbl3.mta_tax |
test_tbl3.tip_amount | test_tbl3.tolls_amount |
test_tbl3.improvement_surcharge | test_tbl3.total_amount |
test_tbl3.congestion_surcharge | test_tbl3.date_col |
test_tbl3._hoodie_partition_path |
+--------------------------------+---------------------------------+-------------------------------+----------------------------------------------------+---------------------+---------------------------------+----------------------------------+----------------------------+--------------------------+-----------------------+-------------------------------+-------------------------+-------------------------+-------------------------+------------------------+------------------+--------------------+-----------------------+-------------------------+----------------------------------+-------------------------+---------------------------------+---------------------+-----------------------------------+
| 20220415180627021 | 20220415180627021_7_1085992 | 2008-12-31
23:02:59 |
e78169d4-03a8-40e0-ad11-9ae43a52b565-0_7-155-6608_20220415180627021.parquet | 2
| 2008-12-31 23:02:59 | 2009-01-01 18:22:41
| 1 | 0.99 | 1
| N | 249 | 90
| 2 | 7.0 | 1.0
| 0.5 | 0.0 | 0.0 | 0.3
| 11.3 | 2.5
| 2008-12-31 | 2008-12-31 |
| 20220415180627021 | 20220415180627021_7_1085996 | 2008-12-31
23:07:03 |
e78169d4-03a8-40e0-ad11-9ae43a52b565-0_7-155-6608_20220415180627021.parquet | 2
| 2008-12-31 23:07:03 | 2008-12-31 23:19:26
| 1 | 1.39 | 1
| N | 107 | 162
| 2 | 8.5 | 0.0
| 0.5 | 0.0 | 0.0 | 0.3
| 11.8 | 2.5
| 2008-12-31 | 2008-12-31 |
| 20220415180627021 | 20220415180627021_7_1085998 | 2008-12-31
23:43:51 |
e78169d4-03a8-40e0-ad11-9ae43a52b565-0_7-155-6608_20220415180627021.parquet | 2
| 2008-12-31 23:43:51 | 2009-01-01 10:32:34
| 1 | 0.79 | 1
| N | 170 | 264
| 1 | 9.5 | 2.5
| 0.5 | 2.56 | 0.0 | 0.3
| 15.36 | 0.0
| 2008-12-31 | 2008-12-31 |
| 20220415180627021 | 20220415180627021_7_1086000 | 2008-12-31
23:03:52 |
e78169d4-03a8-40e0-ad11-9ae43a52b565-0_7-155-6608_20220415180627021.parquet | 2
| 2008-12-31 23:03:52 | 2008-12-31 23:22:12
| 1 | 2.42 | 1
| N | 107 | 237
| 2 | 14.0 | 0.0
| 0.5 | 0.0 | 0.0 | 0.3
| 17.3 | 2.5
| 2008-12-31 | 2008-12-31 |
| 20220415180627021 | 20220415180627021_0_856333 | 2009-01-01
00:02:19 |
f4cb2f09-25c9-48ef-8c81-7b58156f397a-0_0-155-6601_20220415180627021.parquet | 2
| 2009-01-01 00:02:19 | 2009-01-01 09:43:46
| 1 | 1.86 | 1
| N | 45 | 107
| 2 | 10.0 | 0.0
| 0.5 | 0.0 | 0.0 | 0.3
| 13.3 | 2.5
| 2009-01-01 | 2009-01-01 |
+--------------------------------+---------------------------------+-------------------------------+----------------------------------------------------+---------------------+---------------------------------+----------------------------------+----------------------------+--------------------------+-----------------------+-------------------------------+-------------------------+-------------------------+-------------------------+------------------------+------------------+--------------------+-----------------------+-------------------------+----------------------------------+-------------------------+---------------------------------+---------------------+-----------------------------------+
{code}
> Run latest hudi w/ EMR spark and report to aws folks
> ----------------------------------------------------
>
> Key: HUDI-3749
> URL: https://issues.apache.org/jira/browse/HUDI-3749
> Project: Apache Hudi
> Issue Type: Task
> Components: tests-ci
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Blocker
> Fix For: 0.11.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
