[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770080#comment-17770080
]
Jayadeep Jayaraman commented on FLINK-33018:
Hi [~rskraba] - I looked at the PR and tested it as well and it looks good to
me.
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Critical
> Labels: auto-deprioritized-blocker, pull-request-available
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767976#comment-17767976
]
Ryan Skraba commented on FLINK-33018:
-
Hey [~jjayadeep], would you like to take a look at the PR? It's not likely
that it's related to the version bumps, but there's a lack of eyes in the
Pub/Sub area :D Any review would be appreciated!
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Critical
> Labels: auto-deprioritized-blocker, pull-request-available
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764749#comment-17764749
]
Ryan Skraba commented on FLINK-33018:
-
I don't think this JIRA is necessarily a blocker -- it looks like an error in
the test that shows up as flakiness (as opposed to an error in the connector).
The above PR fixes the test, if anybody wants to take a look!
The "eventual segfault" that I noticed early happens before and after the fix,
and is *very likely* a different error due to the
{{MockStreamingRuntimeContext}} creating and not closing a {{MockEnvironment}}.
We could probably fix this, but it should never occur unless you try and run
one of these tests 100K times! What do you think?
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
> Labels: pull-request-available, stale-blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762793#comment-17762793
]
Ryan Skraba commented on FLINK-33018:
-
I think I have a fix -- it looks like occasionally (every 10K runs or so) we
hit the explicit cancel() written in this specific test, before we hit the
implicit cancel() due to the endofStream message being discovered. I'm letting
this run for a while before submitting a PR. This test also slows down in
consecutive runs, which might indicate a leak but I'm hoping this is not the
case.
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762765#comment-17762765
]
Jayadeep Jayaraman commented on FLINK-33018:
I ran it 23K times yesterday and didnt face any issue hence stopped the test
:). Good that you were able to reproduce the error!
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762728#comment-17762728
]
Ryan Skraba commented on FLINK-33018:
-
Hey, I can reproduce this -- all you need to do is run this test about 12,000
times ;) (using IntelliJ repeat until fail).
{code}
Connected to the target VM, address: '127.0.0.1:44879', transport: 'socket'
org.opentest4j.AssertionFailedError:
expected: ["1", "2", "3"]
but was: ["1", "2"]
Expected :["1", "2", "3"]
Actual :["1", "2"]
{code}
I'm taking a look.
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761832#comment-17761832
]
Ryan Skraba commented on FLINK-33018:
-
_If I understand correctly_ -- the *"C"* element is meant to be acked but not
emitted in this test. This (third) end-of-stream message indicates that
processing is finished, so it _should_ have two emitted elements (A and B) but
three acknowledged ids (1, 2, and 3).
I'm reading through the code, and I don't believe there's any relevant changes
due to the bom bump. Is this always reproducible in the build environment or
is it flaky? If flaky -- how flaky? Often or rarely?
I don't have any good hypothesis for the moment. Maybe there's a
synchronization issue on the private test class [collecting the
acks|https://github.com/apache/flink-connector-gcp-pubsub/blob/c06e25a97c5f74873db9d272a4f2cf4787185c4a/flink-connector-gcp-pubsub/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubConsumingTest.java#L214]?
That doesn't seem quite right, since the {{notifyCheckpointComplete}} should
have [already
occurred|https://github.com/apache/flink-connector-gcp-pubsub/blob/c06e25a97c5f74873db9d272a4f2cf4787185c4a/flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/AcknowledgeOnCheckpoint.java#L84]
just before the assert.
Any other clues that might help narrow this down?
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761774#comment-17761774
]
Jayadeep Jayaraman commented on FLINK-33018:
I checked and looks like the test was flaky, I re-ran it in my environment and
it passed successfully. Can you please re-run the test
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761768#comment-17761768
]
Jayadeep Jayaraman commented on FLINK-33018:
Let me look into it and raise a PR
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761762#comment-17761762
]
Martijn Visser commented on FLINK-33018:
[~jjayadeep] I don't know, if you think that's the fix then more then happy to
get it merged :)
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761756#comment-17761756
]
Jayadeep Jayaraman commented on FLINK-33018:
[~martijnvisser] - Shouldn't the assert just check for ("A","B") as mentioned -
[https://github.com/apache/flink-connector-gcp-pubsub/blame/4fdaca7b42969d19bd939c0823afb5372c51461b/flink-connector-gcp-pubsub/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubConsumingTest.java#L109-L119C62]
or change the assert to include Array List to ("A","B","C")
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761711#comment-17761711
]
Martijn Visser commented on FLINK-33018:
[~jjayadeep] [~rskraba] Any thoughts on what might cause this? The only update
that was merged recently was FLINK-32031
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Blocker
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
