[
https://issues.apache.org/jira/browse/HUDI-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479629#comment-17479629
]
sivabalan narayanan commented on HUDI-3262:
-------------------------------------------
I tested integ test bundle for spark2 and its all good w/ latest master.
{code:java}
.
.
.
22/01/20 19:30:24 INFO DagScheduler: Finished executing
7cbb0f66-6b04-40b7-a471-ece73a2a8779
22/01/20 19:30:24 WARN DagScheduler: Executing node "second_insert" ::
{"name":"95c3111e-30d6-4c6d-b55b-544d0e5dd9aa","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":10000,"config":"second_insert"}
22/01/20 19:30:24 INFO DagNode: Generating input data for node
95c3111e-30d6-4c6d-b55b-544d0e5dd9aa
22/01/20 19:30:24 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:30:25 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/2/f48b4f6b-8957-4346-8c66-945b67063954.avro
22/01/20 19:30:25 INFO DagNode: Configs :
{"name":"95c3111e-30d6-4c6d-b55b-544d0e5dd9aa","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":10000,"config":"second_insert"}
22/01/20 19:30:25 INFO DagNode: Inserting input data
95c3111e-30d6-4c6d-b55b-544d0e5dd9aa
22/01/20 19:30:25 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector,
checkpoint: Option{val=1} sourceLimit: 9223372036854775807 lastBatchId: 1
nextBatchId: 2
00:10 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
00:11 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
00:13 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:30:31 INFO DagScheduler: Finished executing
95c3111e-30d6-4c6d-b55b-544d0e5dd9aa
22/01/20 19:30:31 WARN DagScheduler: Executing node "third_insert" ::
{"name":"965a7f10-cfaa-4086-9233-386ab5ca8086","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":300,"config":"third_insert"}
22/01/20 19:30:31 INFO DagNode: Generating input data for node
965a7f10-cfaa-4086-9233-386ab5ca8086
22/01/20 19:30:31 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:30:31 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/3/d20d3a9b-dea3-431e-aa3f-ee203f716b6f.avro
22/01/20 19:30:31 INFO DagNode: Configs :
{"name":"965a7f10-cfaa-4086-9233-386ab5ca8086","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":300,"config":"third_insert"}
22/01/20 19:30:31 INFO DagNode: Inserting input data
965a7f10-cfaa-4086-9233-386ab5ca8086
22/01/20 19:30:31 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector,
checkpoint: Option{val=2} sourceLimit: 9223372036854775807 lastBatchId: 2
nextBatchId: 3
00:15 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
00:16 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
00:17 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:30:36 INFO DagScheduler: Finished executing
965a7f10-cfaa-4086-9233-386ab5ca8086
22/01/20 19:30:36 WARN DagScheduler: Executing node "first_hive_sync" ::
{"queue_name":"adhoc","engine":"mr","name":"427549c7-65f3-484d-8e85-7cf4adfb4b77","config":"first_hive_sync"}
22/01/20 19:30:36 INFO DagNode: Executing hive sync node
22/01/20 19:30:41 INFO DagScheduler: Finished executing
427549c7-65f3-484d-8e85-7cf4adfb4b77
22/01/20 19:30:41 WARN DagScheduler: Executing node "first_validate" ::
{"name":"61f58646-16be-49ce-8695-bdf5c49f1f31","validate_hive":false,"config":"first_validate"}
22/01/20 19:30:41 WARN DagNode: Validation using data from input path
/user/hive/warehouse/hudi-integ-test-suite/input/*/*
22/01/20 19:30:45 INFO ValidateDatasetNode: Validate data in target hudi path
/user/hive/warehouse/hudi-integ-test-suite/output/*/*/*
22/01/20 19:31:01 INFO DagScheduler: Finished executing
61f58646-16be-49ce-8695-bdf5c49f1f31
22/01/20 19:31:01 WARN DagScheduler: Executing node "first_upsert" ::
{"name":"dcbd7071-f044-4a3a-b245-af5b873e5860","record_size":1000,"repeat_count":1,"num_records_upsert":100,"num_partitions_insert":1,"num_records_insert":300,"num_partitions_upsert":1,"config":"first_upsert"}
22/01/20 19:31:01 INFO DagNode: Generating input data
dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:01 INFO DFSHoodieDatasetInputReader: NumPartitions :
Option{val=1}, NumFiles : Option{val=1}, numRecordsToUpdate : Option{val=100},
percentageRecordsPerFile : Optional.empty
22/01/20 19:31:01 INFO DFSHoodieDatasetInputReader: Finished generating updates
22/01/20 19:31:01 INFO DeltaGenerator: Repartitioning records into 1 partitions
for updates
22/01/20 19:31:01 INFO DeltaGenerator: Repartitioning records done for updates
22/01/20 19:31:02 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:31:02 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:31:02 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/4/b3966bb1-e279-42e9-ba9f-fdd61afedc18.avro
22/01/20 19:31:02 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/4/8c455a6e-f514-4640-97a3-11a701fb3a55.avro
22/01/20 19:31:02 INFO DagNode: Configs :
{"name":"dcbd7071-f044-4a3a-b245-af5b873e5860","record_size":1000,"repeat_count":1,"num_records_upsert":100,"num_partitions_insert":1,"num_records_insert":300,"num_partitions_upsert":1,"config":"first_upsert"}
22/01/20 19:31:02 INFO DagNode: Inserting input data
dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:02 INFO DagNode: Upserting input data
dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:02 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector,
checkpoint: Option{val=3} sourceLimit: 9223372036854775807 lastBatchId: 3
nextBatchId: 4
00:51 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
00:52 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
00:54 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:31:12 INFO DagScheduler: Finished executing
dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:12 WARN DagScheduler: Executing node "first_delete" ::
{"name":"053b3a65-737c-4afd-bb52-6e009a5a5179","num_partitions_delete":1,"num_records_delete":2000,"config":"first_delete"}
22/01/20 19:31:12 INFO DFSHoodieDatasetInputReader: NumPartitions :
Option{val=1}, NumFiles : Option{val=1}, numRecordsToUpdate : Option{val=2000},
percentageRecordsPerFile : Optional.empty
22/01/20 19:31:13 INFO DFSHoodieDatasetInputReader: Finished generating updates
22/01/20 19:31:13 INFO DeltaGenerator: Repartitioning records for delete
22/01/20 19:31:13 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1024
22/01/20 19:31:13 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1024
22/01/20 19:31:13 WARN GenericRecordFullPayloadGenerator: The schema does not
have any collections/complex fields. Cannot achieve minPayloadSize : 1024
22/01/20 19:31:14 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/5/3efade2a-70e2-454a-a8a9-5f253cd306df.avro
22/01/20 19:31:14 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/5/58d7c3fb-c5fa-4d9a-93db-fc00b18dbf9e.avro
22/01/20 19:31:14 INFO AvroFileDeltaInputWriter: New Avro File :
hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/5/1c102439-a98a-45a7-98a7-44f0c167a4b6.avro
22/01/20 19:31:14 INFO DagNode: Configs :
{"name":"053b3a65-737c-4afd-bb52-6e009a5a5179","num_partitions_delete":1,"num_records_delete":2000,"config":"first_delete"}
22/01/20 19:31:14 INFO DagNode: Inserting input data
053b3a65-737c-4afd-bb52-6e009a5a5179
22/01/20 19:31:14 INFO DagNode: Deleting input data
053b3a65-737c-4afd-bb52-6e009a5a5179
22/01/20 19:31:14 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector,
checkpoint: Option{val=4} sourceLimit: 9223372036854775807 lastBatchId: 4
nextBatchId: 5
01:04 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
01:05 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
01:08 WARN: Timeline-server-based markers are configured as the marker type
but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:31:26 INFO DagScheduler: Finished executing
053b3a65-737c-4afd-bb52-6e009a5a5179
22/01/20 19:31:26 WARN DagScheduler: Executing node "second_hive_sync" ::
{"queue_name":"adhoc","engine":"mr","name":"dcb5e6c7-a6a4-469d-9510-bfdacc240b04","config":"second_hive_sync"}
22/01/20 19:31:26 INFO DagNode: Executing hive sync node
22/01/20 19:31:27 INFO DagScheduler: Finished executing
dcb5e6c7-a6a4-469d-9510-bfdacc240b04
22/01/20 19:31:27 WARN DagScheduler: Executing node "second_validate" ::
{"delete_input_data":true,"name":"12a098f1-0a27-45c2-ad2c-86012d8f9106","validate_hive":false,"config":"second_validate"}
22/01/20 19:31:27 WARN DagNode: Validation using data from input path
/user/hive/warehouse/hudi-integ-test-suite/input/*/*
22/01/20 19:31:27 INFO ValidateDatasetNode: Validate data in target hudi path
/user/hive/warehouse/hudi-integ-test-suite/output/*/*/*
22/01/20 19:31:38 INFO DagScheduler: Finished executing
12a098f1-0a27-45c2-ad2c-86012d8f9106
22/01/20 19:31:38 INFO DagScheduler: Finished workloads for round num 1
22/01/20 19:31:38 INFO DagScheduler: Finished workloads
22/01/20 19:31:38 INFO HoodieTestSuiteJob: Finished scheduling all tasks, Time
taken 88385{code}
> Integration test suite failure
> ------------------------------
>
> Key: HUDI-3262
> URL: https://issues.apache.org/jira/browse/HUDI-3262
> Project: Apache Hudi
> Issue Type: Bug
> Components: tests-ci
> Reporter: Raymond Xu
> Assignee: sivabalan narayanan
> Priority: Critical
> Labels: sev:normal
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> detailed in https://github.com/apache/hudi/issues/4621
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
