[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440864#comment-17440864
]
Raymond Xu commented on HUDI-2077:
--
Sometimes ran into timeout due to being stuck for long time
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=3190=logs=3272dbb2-0925-5f35-bae7-04e75ae62175=fb428e45-27ff-524a-7e12-db1cb49c418a
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Sagar Sumit
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.10.0
>
> Attachments: 28.txt, hudi_2077_schema_mismatch.txt
>
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
> Search "testUpsertsMORContinuousModeWithMultipleWriters" in the log file for
> details.
> {quote}
> 1730667 [pool-1461-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : }}
> org.apache.hudi.exception.HoodieIOException: Could not check if
> hdfs://localhost:4/user/vsts/continuous_mor_mulitwriter is a valid table
> at
> org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:112)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:73)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:606)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:322)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer.lambda$runJobsInParallel$8(TestHoodieDeltaStreamer.java:906)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:347)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> {{Caused by: java.net.ConnectException: Call From fv-az238-328/10.1.0.24 to
> localhost:4 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> [http://wiki.apache.org/hadoop/ConnectionRefused]
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437970#comment-17437970
]
Sagar Sumit commented on HUDI-2077:
---
Let's keep it open. I found one failing deltastreamer test in
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=3101=results
That may well be due to the PR. But let's keep it open for a few days. If we
don't see any flakiness, then I'll close it.
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Sagar Sumit
>Priority: Critical
> Labels: pull-request-available
> Attachments: 28.txt, hudi_2077_schema_mismatch.txt
>
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
> Search "testUpsertsMORContinuousModeWithMultipleWriters" in the log file for
> details.
> {quote}
> 1730667 [pool-1461-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : }}
> org.apache.hudi.exception.HoodieIOException: Could not check if
> hdfs://localhost:4/user/vsts/continuous_mor_mulitwriter is a valid table
> at
> org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:112)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:73)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:606)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:322)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer.lambda$runJobsInParallel$8(TestHoodieDeltaStreamer.java:906)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:347)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> {{Caused by: java.net.ConnectException: Call From fv-az238-328/10.1.0.24 to
> localhost:4 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> [http://wiki.apache.org/hadoop/ConnectionRefused]
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437941#comment-17437941
]
sivabalan narayanan commented on HUDI-2077:
---
[~xushiyan] [~codope]: can we close this out or do we have any more pending.
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Sagar Sumit
>Priority: Critical
> Labels: pull-request-available
> Attachments: 28.txt, hudi_2077_schema_mismatch.txt
>
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
> Search "testUpsertsMORContinuousModeWithMultipleWriters" in the log file for
> details.
> {quote}
> 1730667 [pool-1461-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : }}
> org.apache.hudi.exception.HoodieIOException: Could not check if
> hdfs://localhost:4/user/vsts/continuous_mor_mulitwriter is a valid table
> at
> org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:112)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:73)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:606)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:322)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer.lambda$runJobsInParallel$8(TestHoodieDeltaStreamer.java:906)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:347)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> {{Caused by: java.net.ConnectException: Call From fv-az238-328/10.1.0.24 to
> localhost:4 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> [http://wiki.apache.org/hadoop/ConnectionRefused]
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431013#comment-17431013
]
Sagar Sumit commented on HUDI-2077:
---
[^hudi_2077_schema_mismatch.txt]
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Sagar Sumit
>Priority: Major
> Labels: pull-request-available
> Attachments: 28.txt, hudi_2077_schema_mismatch.txt
>
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
> Search "testUpsertsMORContinuousModeWithMultipleWriters" in the log file for
> details.
> {quote}
> 1730667 [pool-1461-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : }}
> org.apache.hudi.exception.HoodieIOException: Could not check if
> hdfs://localhost:4/user/vsts/continuous_mor_mulitwriter is a valid table
> at
> org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:112)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:73)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:606)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:322)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer.lambda$runJobsInParallel$8(TestHoodieDeltaStreamer.java:906)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:347)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> {{Caused by: java.net.ConnectException: Call From fv-az238-328/10.1.0.24 to
> localhost:4 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> [http://wiki.apache.org/hadoop/ConnectionRefused]
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431004#comment-17431004
]
Sagar Sumit commented on HUDI-2077:
---
There's one more flaky test.
*TestHoodieDeltaStreamer.testAsyncClusteringServiceWithCompaction*
https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/2725/logs/61
I ran locally 50 times but it never failed. From the CI logs, the reason seems
to be incompatible metadata schema due to a rename. At this point, I am not
sure how the rename happened. The writer schema which is the current schema is
the correct one according to `HoodieMetadataRecord`. For reference I'm
attaching the writer and table schemas.
I am going to disable the scehma validation (anyway this is disabled by
default, but it is hardcoded to true in `HoodieBackedTableMetadataWriter`).
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Raymond Xu
>Priority: Major
> Labels: pull-request-available
> Attachments: 28.txt
>
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
> Search "testUpsertsMORContinuousModeWithMultipleWriters" in the log file for
> details.
> {quote}
> 1730667 [pool-1461-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error : }}
> org.apache.hudi.exception.HoodieIOException: Could not check if
> hdfs://localhost:4/user/vsts/continuous_mor_mulitwriter is a valid table
> at
> org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:112)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:73)
>
> at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:606)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:322)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer.lambda$runJobsInParallel$8(TestHoodieDeltaStreamer.java:906)
>
> at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:347)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> {{Caused by: java.net.ConnectException: Call From fv-az238-328/10.1.0.24 to
> localhost:4 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> [http://wiki.apache.org/hadoop/ConnectionRefused]
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372228#comment-17372228
]
Vinoth Chandar commented on HUDI-2077:
--
[https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=332=logs=6e0b29f5-1de0-523f-3009-f7f76799ff4a=b87cdf6a-7aa9-5ce3-5603-871088f0bd10]
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Sagar Sumit
>Priority: Major
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2077) Flaky test: TestHoodieDeltaStreamer
[
https://issues.apache.org/jira/browse/HUDI-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370132#comment-17370132
]
Raymond Xu commented on HUDI-2077:
--
[~codope] assigning to you since you made a PR to increase the timeout. will
continue to observe it.
> Flaky test: TestHoodieDeltaStreamer
> ---
>
> Key: HUDI-2077
> URL: https://issues.apache.org/jira/browse/HUDI-2077
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: Raymond Xu
>Assignee: Sagar Sumit
>Priority: Major
>
> {code:java}
> [INFO] Results:8520[INFO] 8521[ERROR] Errors: 8522[ERROR]
> TestHoodieDeltaStreamer.testUpsertsMORContinuousModeWithMultipleWriters:716->testUpsertsContinuousModeWithMultipleWriters:831->runJobsInParallel:940
> » Execution{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
