[
https://issues.apache.org/jira/browse/HUDI-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344141#comment-17344141
]
Eldhose Paul edited comment on HUDI-1894 at 5/13/21, 9:32 PM:
--------------------------------------------------------------
[~shivnarayan] I tried in to query this table later for couple more tickets,
Log Files:
{code:java}
Found 40 items
-rwxrwx--x+ 3 hive hive 18213 2021-05-13 17:08
/user/hive/warehouse/jira_expl.db/jiraissue_events/.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513170024.log.1_5-36696-1755439
-rwxrwx--x+ 3 hive hive 3359 2021-05-13 17:04
/user/hive/warehouse/jira_expl.db/jiraissue_events/.3105cc7d-82a5-4d3c-b2fd-76579476abda-0_20210513165035.log.1_4-36862-1763180
-rwxrwx--x+ 3 hive hive 7756 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_20210513170024.log.1_3-37224-1780551
-rwxrwx--x+ 3 hive hive 7767 2021-05-13 17:06
/user/hive/warehouse/jira_expl.db/jiraissue_events/.7a457126-49ed-4a01-a256-c20bfceafa25-0_20210513161037.log.1_2-36862-1763182
-rwxrwx--x+ 3 hive hive 3343 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_20210513165035.log.1_4-37425-1790186
-rwxrwx--x+ 3 hive hive 7514 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.a72267c5-7784-49df-b544-353e1e5877da-0_20210513170621.log.1_5-37224-1780553
-rwxrwx--x+ 3 hive hive 23793 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_20210513170024.log.1_1-36696-1755435
-rwxrwx--x+ 3 hive hive 24511 2021-05-13 17:08
/user/hive/warehouse/jira_expl.db/jiraissue_events/.c087f486-0635-4e81-90d6-5cf330f0dc88-0_20210513170024.log.1_2-36696-1755436
-rwxrwx--x+ 3 hive hive 15707 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_20210513170024.log.1_3-36696-1755437
-rwxrwx--x+ 3 hive hive 12308 2021-05-13 17:08
/user/hive/warehouse/jira_expl.db/jiraissue_events/.e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_20210513170024.log.1_0-36696-1755434
drwxrwx--x+ - hive hive 0 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.hoodie
-rwxrwx--x+ 3 hive hive 93 2021-05-12 10:05
/user/hive/warehouse/jira_expl.db/jiraissue_events/.hoodie_partition_metadata
-rwxrwx--x+ 3 hive hive 1988727 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/031a359b-f8f0-417a-888b-45f2a0b3a26f-0_0-36538-1747717_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 1988831 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/031a359b-f8f0-417a-888b-45f2a0b3a26f-0_2-37435-1790205_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 12432709 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/2d6621ac-7da9-46d7-be43-c4f45f785e11-0_8-36538-1747725_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 26324152 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/3105cc7d-82a5-4d3c-b2fd-76579476abda-0_8-37435-1790211_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 26323905 2021-05-13 16:50
/user/hive/warehouse/jira_expl.db/jiraissue_events/3105cc7d-82a5-4d3c-b2fd-76579476abda-0_9-35706-1708935_20210513165035.parquet
-rwxrwx--x+ 3 hive hive 12315783 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_6-37435-1790209_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 12315541 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_7-36538-1747724_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 24052623 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/433a3869-622c-4ec2-b53e-6658762093af-0_6-36538-1747723_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 502140 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/68d9864f-d9d0-41cd-8e39-60480e723a57-0_0-36528-1747701_20210513170017.parquet
-rwxrwx--x+ 3 hive hive 502766 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/68d9864f-d9d0-41cd-8e39-60480e723a57-0_5-37425-1790187_20210513171034.parquet
-rwxrwx--x+ 3 hive hive 24648164 2021-05-13 16:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/7a457126-49ed-4a01-a256-c20bfceafa25-0_10-32225-1544108_20210513161037.parquet
-rwxrwx--x+ 3 hive hive 24648308 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/7a457126-49ed-4a01-a256-c20bfceafa25-0_5-37435-1790208_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 25891872 2021-05-13 16:50
/user/hive/warehouse/jira_expl.db/jiraissue_events/7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_6-35706-1708932_20210513165035.parquet
-rwxrwx--x+ 3 hive hive 25891900 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_9-37435-1790212_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 500946 2021-05-13 17:02
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_4-36696-1755438_20210513170219.parquet
-rwxrwx--x+ 3 hive hive 501596 2021-05-13 17:06
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_4-37024-1770905_20210513170621.parquet
-rwxrwx--x+ 3 hive hive 501280 2021-05-13 17:04
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_5-36862-1763179_20210513170418.parquet
-rwxrwx--x+ 3 hive hive 501671 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_7-37435-1790210_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 33002191 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_1-37435-1790204_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 32997721 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_2-36538-1747719_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 511943 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/c087f486-0635-4e81-90d6-5cf330f0dc88-0_0-37435-1790203_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 511920 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/c087f486-0635-4e81-90d6-5cf330f0dc88-0_1-36538-1747718_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 11189802 2021-05-13 16:40
/user/hive/warehouse/jira_expl.db/jiraissue_events/cae7231d-8549-423f-8e0f-fd7246fb0780-0_9-34885-1669829_20210513164031.parquet
-rwxrwx--x+ 3 hive hive 752950 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_3-36538-1747720_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 753533 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_3-37435-1790206_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 11560528 2021-05-13 16:50
/user/hive/warehouse/jira_expl.db/jiraissue_events/dcaab0d3-3fa7-4ad3-8765-01c6181fc73d-0_10-35706-1708936_20210513165035.parquet
-rwxrwx--x+ 3 hive hive 15095795 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_4-37435-1790207_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 15095742 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_5-36538-1747722_20210513170024.parquet
{code}
Result: looks good
{code:java}
+-------------------+-------------------------+------------------+----------------------+-----------------------------------------------------------------------------+-----------------------+-------------------+
|_hoodie_commit_time|_hoodie_commit_seqno
|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name
|resolutiondate |archiveddate
|
+-------------------+-------------------------+------------------+----------------------+-----------------------------------------------------------------------------+-----------------------+-------------------+
|20210513104112 |20210513104112_1_371 |1249140.0 |
|031a359b-f8f0-417a-888b-45f2a0b3a26f-0_1-511-25386_20210513104112.parquet
|2021-05-06 16:04:26.074|1969-12-31 19:00:00|
|20210512120157 |20210512120157_10_2599400|1249149.0 |
|031a359b-f8f0-417a-888b-45f2a0b3a26f-0_10-2536-142469_20210512120157.parquet
|2021-05-07 15:44:52.262|1969-12-31 19:00:00|
|20210513143102 |20210513143102_0_4340 |1249148.0 |
|031a359b-f8f0-417a-888b-45f2a0b3a26f-0_0-22210-1061170_20210513143102.parquet|2021-05-13
14:28:29.064|1969-12-31 19:00:00|
|20210512111134 |20210512111134_1_1087908 |1008136.0 |
|39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_1-1558-86150_20210512111134.parquet
|null |null |
|20210512111134 |20210512111134_1_1113271 |1009140.0 |
|39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_1-1558-86150_20210512111134.parquet
|2021-01-08 12:59:24.648|null |
|20210512111134 |20210512111134_1_1131535 |1009137.0 |
|39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_1-1558-86150_20210512111134.parquet
|2021-01-08 12:59:46.932|null |
|20210512144432 |20210512144432_2_113 |1256385.0 |
|dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_2-3726-181165_20210512144432.parquet
|2021-05-12 14:43:31.178|1969-12-31 19:00:00|
|20210512144432 |20210512144432_2_114 |1256388.0 |
|dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_2-3726-181165_20210512144432.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00|
|20210513103921 |20210513103921_7_85 |1256488.0 |
|dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_7-228-11828_20210513103921.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00|
|20210513134240 |20210513134240_8_4104 |1257959.0 |
|68d9864f-d9d0-41cd-8e39-60480e723a57-0_8-17689-844700_20210513134240.parquet
|null |null |
|20210513134432 |20210513134432_2_3865 |1257967.0 |
|68d9864f-d9d0-41cd-8e39-60480e723a57-0_2-17889-854376_20210513134432.parquet
|null |null |
|20210513134824 |20210513134824_2_2342 |1257974.0 |
|68d9864f-d9d0-41cd-8e39-60480e723a57-0_2-18250-871764_20210513134824.parquet
|null |null |
|20210513135054 |20210513135054_4_40220 |1257882.0 |
|c087f486-0635-4e81-90d6-5cf330f0dc88-0_4-18464-881489_20210513135054.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00|
|20210513135232 |20210513135232_2_4107 |1257981.0 |
|c087f486-0635-4e81-90d6-5cf330f0dc88-0_2-18659-891200_20210513135232.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00|
|20210513140848 |20210513140848_3_4428 |1258041.0 |
|c087f486-0635-4e81-90d6-5cf330f0dc88-0_3-19975-953464_20210513140848.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00|
|20210512104547 |20210512104547_0_408297 |1013949.0 |
|3105cc7d-82a5-4d3c-b2fd-76579476abda-0_0-1069-58132_20210512104547.parquet
|2020-12-18 09:42:58.023|1969-12-31 19:00:00|
|20210512104547 |20210512104547_0_448903 |1008116.0 |
|3105cc7d-82a5-4d3c-b2fd-76579476abda-0_0-1069-58132_20210512104547.parquet
|2019-07-05 06:43:10.899|1969-12-31 19:00:00|
|20210512104547 |20210512104547_0_450309 |1008113.0 |
|3105cc7d-82a5-4d3c-b2fd-76579476abda-0_0-1069-58132_20210512104547.parquet
|2019-07-29 09:40:38.163|1969-12-31 19:00:00|
|20210512113847 |20210512113847_9_2650436 |1016123.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-11-23 08:59:39.771|1969-12-31 19:00:00|
|20210512113847 |20210512113847_9_2677320 |1007370.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-12-10 09:52:44.083|1969-12-31 19:00:00|
|20210512113847 |20210512113847_9_2687968 |1008036.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-11-26 12:15:56.002|1969-12-31 19:00:00|
|20210512113847 |20210512113847_9_2698975 |1007378.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-12-10 09:53:17.318|1969-12-31 19:00:00|
|20210512111134 |20210512111134_0_1133961 |1007376.0 |
|7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_0-1558-86149_20210512111134.parquet
|2020-12-01 01:32:40.513|null |
|20210512111134 |20210512111134_0_1149599 |1007383.0 |
|7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_0-1558-86149_20210512111134.parquet
|null |null |
|20210512111134 |20210512111134_0_1218090 |1007388.0 |
|7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_0-1558-86149_20210512111134.parquet
|2020-12-10 09:52:55.146|null |
|20210512113847 |20210512113847_8_7827892 |1199943.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2021-03-23 10:09:03.755|1969-12-31 19:00:00|
|20210512113847 |20210512113847_8_7828005 |1139941.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-12-08 14:19:21 |1969-12-31 19:00:00|
|20210512121616 |20210512121616_0_3816214 |1237858.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_0-3571-199593_20210512121616.parquet
|2021-01-29 11:42:36.324|1969-12-31 19:00:00|
|20210512113847 |20210512113847_8_7862709 |1007197.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-11-12 08:39:28.371|1969-12-31 19:00:00|
|20210512113847 |20210512113847_8_7884624 |1007262.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-11-12 20:59:17.714|1969-12-31 19:00:00|
|20210512113847 |20210512113847_8_7911459 |1007347.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-11-23 14:46:06.047|1969-12-31 19:00:00|
|20210513121620 |20210513121620_4_1157 |1257851.0 |
|a72267c5-7784-49df-b544-353e1e5877da-0_4-9811-471198_20210513121620.parquet
|null |null |
|20210513121620 |20210513121620_4_1164 |1257857.0 |
|a72267c5-7784-49df-b544-353e1e5877da-0_4-9811-471198_20210513121620.parquet
|null |null |
|20210513145047 |20210513145047_5_4786 |1258085.0 |
|a72267c5-7784-49df-b544-353e1e5877da-0_5-24183-1156010_20210513145047.parquet|null
|null |
|20210512104547 |20210512104547_1_740262 |1214786.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|2021-04-28 22:24:29.054|null |
|20210512104547 |20210512104547_1_750259 |1016047.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|null |null |
|20210512104547 |20210512104547_1_756330 |1008042.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|2020-11-17 10:36:50.987|null |
|20210512104547 |20210512104547_1_774615 |1007909.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|null |null |
|20210513143102 |20210513143102_8_41196 |1242436.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_8-22210-1061178_20210513143102.parquet|2021-05-13
14:28:49.812|null |
+-------------------+-------------------------+------------------+----------------------+-----------------------------------------------------------------------------+-----------------------+-------------------+
{code}
Log File:
ps: The tickets I am querying is not present in log file
{code:java}
Found 45 items
-rwxrwx--x+ 3 hive hive 18213 2021-05-13 17:08
/user/hive/warehouse/jira_expl.db/jiraissue_events/.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513170024.log.1_5-36696-1755439
-rwxrwx--x+ 3 hive hive 3896 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513171040.log.1_5-37626-1799835
-rwxrwx--x+ 3 hive hive 3359 2021-05-13 17:04
/user/hive/warehouse/jira_expl.db/jiraissue_events/.3105cc7d-82a5-4d3c-b2fd-76579476abda-0_20210513165035.log.1_4-36862-1763180
-rwxrwx--x+ 3 hive hive 7756 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_20210513170024.log.1_3-37224-1780551
-rwxrwx--x+ 3 hive hive 4077 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.68d9864f-d9d0-41cd-8e39-60480e723a57-0_20210513171034.log.1_2-37626-1799831
-rwxrwx--x+ 3 hive hive 7767 2021-05-13 17:06
/user/hive/warehouse/jira_expl.db/jiraissue_events/.7a457126-49ed-4a01-a256-c20bfceafa25-0_20210513161037.log.1_2-36862-1763182
-rwxrwx--x+ 3 hive hive 3343 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_20210513165035.log.1_4-37425-1790186
-rwxrwx--x+ 3 hive hive 4252 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_20210513171040.log.1_4-37626-1799834
-rwxrwx--x+ 3 hive hive 7514 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.a72267c5-7784-49df-b544-353e1e5877da-0_20210513170621.log.1_5-37224-1780553
-rwxrwx--x+ 3 hive hive 3394 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.a72267c5-7784-49df-b544-353e1e5877da-0_20210513171040.log.1_3-37626-1799833
-rwxrwx--x+ 3 hive hive 23793 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_20210513170024.log.1_1-36696-1755435
-rwxrwx--x+ 3 hive hive 3946 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_20210513171040.log.1_0-37626-1799830
-rwxrwx--x+ 3 hive hive 24511 2021-05-13 17:08
/user/hive/warehouse/jira_expl.db/jiraissue_events/.c087f486-0635-4e81-90d6-5cf330f0dc88-0_20210513170024.log.1_2-36696-1755436
-rwxrwx--x+ 3 hive hive 6281 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.c087f486-0635-4e81-90d6-5cf330f0dc88-0_20210513171040.log.1_1-37626-1799832
-rwxrwx--x+ 3 hive hive 15707 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/.dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_20210513170024.log.1_3-36696-1755437
-rwxrwx--x+ 3 hive hive 12308 2021-05-13 17:08
/user/hive/warehouse/jira_expl.db/jiraissue_events/.e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_20210513170024.log.1_0-36696-1755434
drwxrwx--x+ - hive hive 0 2021-05-13 17:12
/user/hive/warehouse/jira_expl.db/jiraissue_events/.hoodie
-rwxrwx--x+ 3 hive hive 93 2021-05-12 10:05
/user/hive/warehouse/jira_expl.db/jiraissue_events/.hoodie_partition_metadata
-rwxrwx--x+ 3 hive hive 1988727 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/031a359b-f8f0-417a-888b-45f2a0b3a26f-0_0-36538-1747717_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 1988831 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/031a359b-f8f0-417a-888b-45f2a0b3a26f-0_2-37435-1790205_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 12432709 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/2d6621ac-7da9-46d7-be43-c4f45f785e11-0_8-36538-1747725_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 26324152 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/3105cc7d-82a5-4d3c-b2fd-76579476abda-0_8-37435-1790211_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 26323905 2021-05-13 16:50
/user/hive/warehouse/jira_expl.db/jiraissue_events/3105cc7d-82a5-4d3c-b2fd-76579476abda-0_9-35706-1708935_20210513165035.parquet
-rwxrwx--x+ 3 hive hive 12315783 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_6-37435-1790209_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 12315541 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_7-36538-1747724_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 24052623 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/433a3869-622c-4ec2-b53e-6658762093af-0_6-36538-1747723_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 502140 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/68d9864f-d9d0-41cd-8e39-60480e723a57-0_0-36528-1747701_20210513170017.parquet
-rwxrwx--x+ 3 hive hive 502766 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/68d9864f-d9d0-41cd-8e39-60480e723a57-0_5-37425-1790187_20210513171034.parquet
-rwxrwx--x+ 3 hive hive 24648164 2021-05-13 16:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/7a457126-49ed-4a01-a256-c20bfceafa25-0_10-32225-1544108_20210513161037.parquet
-rwxrwx--x+ 3 hive hive 24648308 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/7a457126-49ed-4a01-a256-c20bfceafa25-0_5-37435-1790208_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 25891872 2021-05-13 16:50
/user/hive/warehouse/jira_expl.db/jiraissue_events/7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_6-35706-1708932_20210513165035.parquet
-rwxrwx--x+ 3 hive hive 25891900 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_9-37435-1790212_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 501596 2021-05-13 17:06
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_4-37024-1770905_20210513170621.parquet
-rwxrwx--x+ 3 hive hive 501280 2021-05-13 17:04
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_5-36862-1763179_20210513170418.parquet
-rwxrwx--x+ 3 hive hive 501671 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/a72267c5-7784-49df-b544-353e1e5877da-0_7-37435-1790210_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 33002191 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_1-37435-1790204_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 32997721 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_2-36538-1747719_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 511943 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/c087f486-0635-4e81-90d6-5cf330f0dc88-0_0-37435-1790203_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 511920 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/c087f486-0635-4e81-90d6-5cf330f0dc88-0_1-36538-1747718_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 11189802 2021-05-13 16:40
/user/hive/warehouse/jira_expl.db/jiraissue_events/cae7231d-8549-423f-8e0f-fd7246fb0780-0_9-34885-1669829_20210513164031.parquet
-rwxrwx--x+ 3 hive hive 752950 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_3-36538-1747720_20210513170024.parquet
-rwxrwx--x+ 3 hive hive 753533 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_3-37435-1790206_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 11560528 2021-05-13 16:50
/user/hive/warehouse/jira_expl.db/jiraissue_events/dcaab0d3-3fa7-4ad3-8765-01c6181fc73d-0_10-35706-1708936_20210513165035.parquet
-rwxrwx--x+ 3 hive hive 15095795 2021-05-13 17:10
/user/hive/warehouse/jira_expl.db/jiraissue_events/e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_4-37435-1790207_20210513171040.parquet
-rwxrwx--x+ 3 hive hive 15095742 2021-05-13 17:00
/user/hive/warehouse/jira_expl.db/jiraissue_events/e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_5-36538-1747722_20210513170024.parquet
{code}
Result: Null replaced by some default values for records which has two instance
of log file.
eg:
.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513170024.log.1_5-36696-1755439
.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513171040.log.1_5-37626-1799835
result from files *031a359b-f8f0-417a-888b-45f2a0b3a26f* are incorrect.
logfiles .*dbf01b8b-98c0-4768-a8d3-f562c5d17a6b.*,
.e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0.
*.3105cc7d-82a5-4d3c-b2fd-76579476abda-0.** only have one entry and
*.2d6621ac-7da9-46d7-be43-c4f45f785e11** doesn't have an entry. Records from
this files looks good
{code:java}
+-------------------+-------------------------+------------------+----------------------+-----------------------------------------------------------------------------+-----------------------+-----------------------+
|_hoodie_commit_time|_hoodie_commit_seqno
|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name
|resolutiondate |archiveddate
|
+-------------------+-------------------------+------------------+----------------------+-----------------------------------------------------------------------------+-----------------------+-----------------------+
|20210513121620 |20210513121620_4_1157 |1257851.0 |
|a72267c5-7784-49df-b544-353e1e5877da-0_4-9811-471198_20210513121620.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210513121620 |20210513121620_4_1164 |1257857.0 |
|a72267c5-7784-49df-b544-353e1e5877da-0_4-9811-471198_20210513121620.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210513145047 |20210513145047_5_4786 |1258085.0 |
|a72267c5-7784-49df-b544-353e1e5877da-0_5-24183-1156010_20210513145047.parquet|1969-12-31
19:00:00 |1969-12-31 19:00:00 |
|20210512113847 |20210512113847_8_7827892 |1199943.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2021-03-23 10:09:03.755|1969-12-31 19:00:00 |
|20210512113847 |20210512113847_8_7828005 |1139941.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-12-08 14:19:21 |1969-12-31 19:00:00 |
|20210512121616 |20210512121616_0_3816214 |1237858.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_0-3571-199593_20210512121616.parquet
|2021-01-29 11:42:36.324|1969-12-31 19:00:00 |
|20210512113847 |20210512113847_8_7862709 |1007197.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-11-12 08:39:28.371|1969-12-31 19:00:00 |
|20210512113847 |20210512113847_8_7884624 |1007262.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-11-12 20:59:17.714|1969-12-31 19:00:00 |
|20210512113847 |20210512113847_8_7911459 |1007347.0 |
|bae6f29e-9458-4e34-a3b2-570db49b9cf1-0_8-2047-114256_20210512113847.parquet
|2020-11-23 14:46:06.047|1969-12-31 19:00:00 |
|20210513134240 |20210513134240_8_4104 |1257959.0 |
|68d9864f-d9d0-41cd-8e39-60480e723a57-0_8-17689-844700_20210513134240.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210513134432 |20210513134432_2_3865 |1257967.0 |
|68d9864f-d9d0-41cd-8e39-60480e723a57-0_2-17889-854376_20210513134432.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210513134824 |20210513134824_2_2342 |1257974.0 |
|68d9864f-d9d0-41cd-8e39-60480e723a57-0_2-18250-871764_20210513134824.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210512104547 |20210512104547_0_408297 |1013949.0 |
|3105cc7d-82a5-4d3c-b2fd-76579476abda-0_0-1069-58132_20210512104547.parquet
|2020-12-18 09:42:58.023|null |
|20210512104547 |20210512104547_0_448903 |1008116.0 |
|3105cc7d-82a5-4d3c-b2fd-76579476abda-0_0-1069-58132_20210512104547.parquet
|null |null |
|20210512104547 |20210512104547_0_450309 |1008113.0 |
|3105cc7d-82a5-4d3c-b2fd-76579476abda-0_0-1069-58132_20210512104547.parquet
|null |null |
|20210512111134 |20210512111134_0_1133961 |1007376.0 |
|7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_0-1558-86149_20210512111134.parquet
|2020-12-01 01:32:40.513|1969-12-31 19:00:00 |
|20210512111134 |20210512111134_0_1149599 |1007383.0 |
|7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_0-1558-86149_20210512111134.parquet
|2020-05-27 11:15:08.398|2021-01-08 12:35:23.076|
|20210512111134 |20210512111134_0_1218090 |1007388.0 |
|7cf9fc72-7612-4ec6-8df3-e41086cea8f2-0_0-1558-86149_20210512111134.parquet
|2020-12-10 09:52:55.146|1969-12-31 19:00:00 |
|20210513135054 |20210513135054_4_40220 |1257882.0 |
|c087f486-0635-4e81-90d6-5cf330f0dc88-0_4-18464-881489_20210513135054.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210513135232 |20210513135232_2_4107 |1257981.0 |
|c087f486-0635-4e81-90d6-5cf330f0dc88-0_2-18659-891200_20210513135232.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210513140848 |20210513140848_3_4428 |1258041.0 |
|c087f486-0635-4e81-90d6-5cf330f0dc88-0_3-19975-953464_20210513140848.parquet
|1969-12-31 19:00:00 |1969-12-31 19:00:00 |
|20210512113847 |20210512113847_9_2650436 |1016123.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-11-23 08:59:39.771|null |
|20210512113847 |20210512113847_9_2677320 |1007370.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-12-10 09:52:44.083|null |
|20210512113847 |20210512113847_9_2687968 |1008036.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-11-26 12:15:56.002|null |
|20210512113847 |20210512113847_9_2698975 |1007378.0 |
|e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0-0_9-2047-114257_20210512113847.parquet
|2020-12-10 09:53:17.318|null |
|20210512111134 |20210512111134_1_1087908 |1008136.0 |
|39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_1-1558-86150_20210512111134.parquet
|null |null |
|20210512111134 |20210512111134_1_1113271 |1009140.0 |
|39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_1-1558-86150_20210512111134.parquet
|2021-01-08 12:59:24.648|null |
|20210512111134 |20210512111134_1_1131535 |1009137.0 |
|39681b5f-33e4-4b51-bc9e-9225d78bbb16-0_1-1558-86150_20210512111134.parquet
|2021-01-08 12:59:46.932|null |
|20210512144432 |20210512144432_2_113 |1256385.0 |
|dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_2-3726-181165_20210512144432.parquet
|2021-05-12 14:43:31.178|null |
|20210512144432 |20210512144432_2_114 |1256388.0 |
|dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_2-3726-181165_20210512144432.parquet
|null |null |
|20210513103921 |20210513103921_7_85 |1256488.0 |
|dbf01b8b-98c0-4768-a8d3-f562c5d17a6b-0_7-228-11828_20210513103921.parquet
|null |null |
|20210513104112 |20210513104112_1_371 |1249140.0 |
|031a359b-f8f0-417a-888b-45f2a0b3a26f-0_1-511-25386_20210513104112.parquet
|2021-05-06 16:04:26.074|1969-12-31 19:00:00 |
|20210512120157 |20210512120157_10_2599400|1249149.0 |
|031a359b-f8f0-417a-888b-45f2a0b3a26f-0_10-2536-142469_20210512120157.parquet
|2021-05-07 15:44:52.262|1969-12-31 19:00:00 |
|20210513143102 |20210513143102_0_4340 |1249148.0 |
|031a359b-f8f0-417a-888b-45f2a0b3a26f-0_0-22210-1061170_20210513143102.parquet|2021-05-13
14:28:29.064|1969-12-31 19:00:00 |
|20210512104547 |20210512104547_1_740262 |1214786.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|2021-04-28 22:24:29.054|null |
|20210512104547 |20210512104547_1_750259 |1016047.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|null |null |
|20210512104547 |20210512104547_1_756330 |1008042.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|2020-11-17 10:36:50.987|null |
|20210512104547 |20210512104547_1_774615 |1007909.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_1-1069-58133_20210512104547.parquet
|null |null |
|20210513143102 |20210513143102_8_41196 |1242436.0 |
|2d6621ac-7da9-46d7-be43-c4f45f785e11-0_8-22210-1061178_20210513143102.parquet|2021-05-13
14:28:49.812|null |
+-------------------+-------------------------+------------------+----------------------+-----------------------------------------------------------------------------+-----------------------+-----------------------+
{code}
was (Author: epaul):
[~shivnarayan] I tried in to query this table later for couple more tickets,
Log Files:
!image-2021-05-13-17-16-20-609.png|width=1054,height=472!
Result: looks good
!image-2021-05-13-17-17-33-181.png!
Log File: and the tickets I am querying is not present in log file
!image-2021-05-13-17-18-43-830.png!
Result: Null replaced by some default values for records which has two instance
of log file.
eg:
.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513170024.log.1_5-36696-1755439
.031a359b-f8f0-417a-888b-45f2a0b3a26f-0_20210513171040.log.1_5-37626-1799835
result from files *031a359b-f8f0-417a-888b-45f2a0b3a26f* are incorrect.
logfiles .*dbf01b8b-98c0-4768-a8d3-f562c5d17a6b.*,
.e54bd2c3-6b9c-4e4b-95d7-0a7b81b83dd0.
*.3105cc7d-82a5-4d3c-b2fd-76579476abda-0.** only have one entry and
*.2d6621ac-7da9-46d7-be43-c4f45f785e11** doesn't have an entry. Records from
this files looks good
!image-2021-05-13-17-24-43-464.png!
> NULL values in timestamp column defaulted
> ------------------------------------------
>
> Key: HUDI-1894
> URL: https://issues.apache.org/jira/browse/HUDI-1894
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
> Reporter: Eldhose Paul
> Assignee: sivabalan narayanan
> Priority: Major
> Labels: sev:critical
> Attachments: image-2021-05-13-17-16-20-609.png,
> image-2021-05-13-17-17-33-181.png, image-2021-05-13-17-18-43-830.png,
> image-2021-05-13-17-24-43-464.png
>
>
> Reading timestamp column from hudi and underlying parquet files in spark
> gives different results.
> *hudi properties:*
> {code:java}
> hdfs dfs -cat
> /user/hive/warehouse/jira_expl.db/jiraissue_events/.hoodie/hoodie.properties
> #Properties saved on Tue May 11 17:17:22 EDT 2021
> #Tue May 11 17:17:22 EDT 2021
> hoodie.compaction.payload.class=org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
> hoodie.table.name=jiraissue
> hoodie.archivelog.folder=archived
> hoodie.table.type=MERGE_ON_READ
> hoodie.table.version=1
> hoodie.timeline.layout.version=1
> {code}
>
> *Reading directly from parquet using Spark:*
> {code:java}
> scala> val ji =
> spark.read.format("parquet").load("/user/hive/warehouse/jira_expl.db/jiraissue_events/*.parquet")
> ji: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string,
> _hoodie_commit_seqno: string ... 49 more fields]scala> ji.filter($"id" ===
> 1237858).withColumn("inputfile",
> input_file_name()).select($"_hoodie_commit_time", $"_hoodie_commit_seqno",
> $"_hoodie_record_key", $"_hoodie_partition_path",
> $"_hoodie_file_name",$"resolutiondate", $"archiveddate",
> $"inputfile").show(false)
> +-------------------+----------------------+------------------+----------------------+-----------------------------------------------------------------------+--------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
> |_hoodie_commit_time|_hoodie_commit_seqno
> |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name
> |resolutiondate|archiveddate|inputfile
>
> |
> +-------------------+----------------------+------------------+----------------------+-----------------------------------------------------------------------+--------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
> |20210511171722 |20210511171722_7_13718|1237858.0 |
>
> |832cf07f-637b-4a4c-ab08-6929554f003a-0_7-98-5106_20210511171722.parquet|null
> |null
> |hdfs://nameservice1/user/hive/warehouse/jira_expl.db/jiraissue_events/832cf07f-637b-4a4c-ab08-6929554f003a-0_7-98-5106_20210511171722.parquet
> |
> |20210511171722 |20210511171722_7_13718|1237858.0 |
>
> |832cf07f-637b-4a4c-ab08-6929554f003a-0_7-98-5106_20210511171722.parquet|null
> |null
> |hdfs://nameservice1/user/hive/warehouse/jira_expl.db/jiraissue_events/832cf07f-637b-4a4c-ab08-6929554f003a-0_8-1610-78711_20210511173615.parquet%7C
> +-------------------+----------------------+------------------+----------------------+-----------------------------------------------------------------------+--------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
> {code}
> *Reading `hudi` using Spark:*
> {code:java}
> scala> val jih =
> spark.read.format("org.apache.hudi").load("/user/hive/warehouse/jira_expl.db/jiraissue_events")
> jih: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string,
> _hoodie_commit_seqno: string ... 49 more fields]scala> jih.filter($"id" ===
> 1237858).select($"_hoodie_commit_time", $"_hoodie_commit_seqno",
> $"_hoodie_record_key", $"_hoodie_partition_path",
> $"_hoodie_file_name",$"resolutiondate", $"archiveddate").show(false)
> +-------------------+----------------------+------------------+----------------------+-----------------------------------------------------------------------+-------------------+-------------------+
> |_hoodie_commit_time|_hoodie_commit_seqno
> |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name
> |resolutiondate |archiveddate |
> +-------------------+----------------------+------------------+----------------------+-----------------------------------------------------------------------+-------------------+-------------------+
> |20210511171722 |20210511171722_7_13718|1237858.0 |
>
> |832cf07f-637b-4a4c-ab08-6929554f003a-0_7-98-5106_20210511171722.parquet|2018-07-30
> 14:58:52|1969-12-31 19:00:00|
> +-------------------+----------------------+------------------+----------------------+-----------------------------------------------------------------------+-------------------+-------------------+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
