[
https://issues.apache.org/jira/browse/HUDI-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balaji Varadarajan updated HUDI-1182:
-------------------------------------
Description:
Documenting all failure cases here :
Integration Tests:
1. ITTestHoodieSanity : Command
([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync,
--table-path, hdfs://namenode/docker_hoodie_single_partition_key_cow_test,
--hive-url, jdbc:hive2://hiveserver:10000, --table-type, COPY_ON_WRITE,
--hive-table, docker_hoodie_single_partition_key_cow_test])
{code:java}
05:17:48.384 [main] ERROR HoodieJavaStreamingApp - Got error running app
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
Expecting 100 records, Got 50 at
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212] at
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212] at
HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:193) ~[test-classes/:?]
at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:126)
[test-classes/:?] Caused by: java.lang.IllegalArgumentException: Expecting 100
records, Got 50 at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
~[hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT] at
HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:352)
~[test-classes/:?] at
HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:186)
~[test-classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_212] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_212] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
{code}
Comments: From the context, it looks like out of 100 records in 1st batch,
only 50 records were updated but all 100 records were expected to be upddated.
2. ITTestHoodieSanity : Command
([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync, --t
able-path, hdfs://namenode/docker_hoodie_single_partition_key_mor_test,
--hive-url, jdbc:hive2://hiveserver:10000, --t
able-type, MERGE_ON_READ, --hive-table,
docker_hoodie_single_partition_key_mor_test]) expected to succeed. Exit (255)
==> expected: <0> but was: <255>
{code:java}
Instants :[[20200812012618__deltacommit__COMPLETED],
[20200812012629__deltacommit__COMPLETED]]
Instants :[[20200812012618__deltacommit__COMPLETED],
[20200812012629__deltacommit__COMPLETED]]
01:29:35.754 [main] ERROR HoodieJavaStreamingApp - Got error running app
java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
Timedout waiting for 3 commits to appear in
hdfs://namenode/docker_hoodie_single_partition_key_mor_test
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212]
at HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:185)
~[test-classes/:?]
at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:118)
[test-classes/:?]
Caused by: java.lang.IllegalStateException: Timedout waiting for 3 commits to
appear in hdfs://namenode/docker_hoodie_
single_partition_key_mor_test
at HoodieJavaStreamingApp.waitTillNCommits(HoodieJavaStreamingApp.java:261)
~[test-classes/:?]
at
HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:298)
~[test-classes/:?]
at HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:178)
~[test-classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]{code}
>From the context: Compaction should have been scheduled and executed but that
>did not happen here.
was:
Documenting all failure cases here :
Integration Tests:
1. ITTestHoodieSanity : Command
([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync,
--table-path, hdfs://namenode/docker_hoodie_single_partition_key_cow_test,
--hive-url, jdbc:hive2://hiveserver:10000, --table-type, COPY_ON_WRITE,
--hive-table, docker_hoodie_single_partition_key_cow_test])
{code:java}
05:17:48.384 [main] ERROR HoodieJavaStreamingApp - Got error running app
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
Expecting 100 records, Got 50 at
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212] at
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212] at
HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:193) ~[test-classes/:?]
at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:126)
[test-classes/:?] Caused by: java.lang.IllegalArgumentException: Expecting 100
records, Got 50 at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
~[hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT] at
HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:352)
~[test-classes/:?] at
HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:186)
~[test-classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_212] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_212] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
{code}
Comments: From the context, it looks like out of 100 records in 1st batch,
only 50 records were updated but all 100 records were expected to be upddated.
2. ITTestHoodieSanity : Command
([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync, --t
able-path, hdfs://namenode/docker_hoodie_single_partition_key_mor_test,
--hive-url, jdbc:hive2://hiveserver:10000, --t
able-type, MERGE_ON_READ, --hive-table,
docker_hoodie_single_partition_key_mor_test]) expected to succeed. Exit (255)
==> expected: <0> but was: <255>
{code:java}
{code}
Instants :[[20200812012618__deltacommit__COMPLETED],
[20200812012629__deltacommit__COMPLETED]]
Instants :[[20200812012618__deltacommit__COMPLETED],
[20200812012629__deltacommit__COMPLETED]]
01:29:35.754 [main] ERROR HoodieJavaStreamingApp - Got error running app
java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
Timedout waiting for 3 commits to appear in
hdfs://namenode/docker_hoodie_single_partition_key_mor_test
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212]
at HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:185)
~[test-classes/:?]
at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:118)
[test-classes/:?]
Caused by: java.lang.IllegalStateException: Timedout waiting for 3 commits to
appear in hdfs://namenode/docker_hoodie_
single_partition_key_mor_test
at HoodieJavaStreamingApp.waitTillNCommits(HoodieJavaStreamingApp.java:261)
~[test-classes/:?]
at
HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:298)
~[test-classes/:?]
at HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:178)
~[test-classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
>From the context: Compaction should have been scheduled and executed but that
>did not happen here.
> Stabilize CI
> -------------
>
> Key: HUDI-1182
> URL: https://issues.apache.org/jira/browse/HUDI-1182
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Testing
> Reporter: Balaji Varadarajan
> Assignee: Balaji Varadarajan
> Priority: Blocker
>
> Documenting all failure cases here :
>
> Integration Tests:
>
> 1. ITTestHoodieSanity : Command
> ([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync,
> --table-path, hdfs://namenode/docker_hoodie_single_partition_key_cow_test,
> --hive-url, jdbc:hive2://hiveserver:10000, --table-type, COPY_ON_WRITE,
> --hive-table, docker_hoodie_single_partition_key_cow_test])
> {code:java}
> 05:17:48.384 [main] ERROR HoodieJavaStreamingApp - Got error running app
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> Expecting 100 records, Got 50 at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212] at
> java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212] at
> HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:193)
> ~[test-classes/:?] at
> HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:126)
> [test-classes/:?] Caused by: java.lang.IllegalArgumentException: Expecting
> 100 records, Got 50 at
> org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
> ~[hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT] at
> HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:352)
> ~[test-classes/:?] at
> HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:186)
> ~[test-classes/:?] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_212] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
> {code}
>
> Comments: From the context, it looks like out of 100 records in 1st batch,
> only 50 records were updated but all 100 records were expected to be upddated.
>
> 2. ITTestHoodieSanity : Command
> ([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync, --t
> able-path, hdfs://namenode/docker_hoodie_single_partition_key_mor_test,
> --hive-url, jdbc:hive2://hiveserver:10000, --t
> able-type, MERGE_ON_READ, --hive-table,
> docker_hoodie_single_partition_key_mor_test]) expected to succeed. Exit (255)
> ==> expected: <0> but was: <255>
>
> {code:java}
> Instants :[[20200812012618__deltacommit__COMPLETED],
> [20200812012629__deltacommit__COMPLETED]]
> Instants :[[20200812012618__deltacommit__COMPLETED],
> [20200812012629__deltacommit__COMPLETED]]
> 01:29:35.754 [main] ERROR HoodieJavaStreamingApp - Got error running app
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> Timedout waiting for 3 commits to appear in
> hdfs://namenode/docker_hoodie_single_partition_key_mor_test
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212]
> at HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:185)
> ~[test-classes/:?]
> at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:118)
> [test-classes/:?]
> Caused by: java.lang.IllegalStateException: Timedout waiting for 3 commits
> to appear in hdfs://namenode/docker_hoodie_
> single_partition_key_mor_test
> at HoodieJavaStreamingApp.waitTillNCommits(HoodieJavaStreamingApp.java:261)
> ~[test-classes/:?]
> at
> HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:298)
> ~[test-classes/:?]
> at HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:178)
> ~[test-classes/:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_212]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_212]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]{code}
>
> From the context: Compaction should have been scheduled and executed but that
> did not happen here.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
