[
https://issues.apache.org/jira/browse/BEAM-13579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491041#comment-17491041
]
Beam JIRA Bot commented on BEAM-13579:
--------------------------------------
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Cannot run python_xlang_kafka_taxi_dataflow validation script on 2.35.0
> -----------------------------------------------------------------------
>
> Key: BEAM-13579
> URL: https://issues.apache.org/jira/browse/BEAM-13579
> Project: Beam
> Issue Type: Bug
> Components: release
> Reporter: Valentyn Tymofieiev
> Assignee: Chamikara Madhusanka Jayalath
> Priority: P1
> Labels: stale-assigned
>
> Dataflow job gets errors like:
> {noformat}
> Error message from worker: generic::unknown:
> org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter
> : failed to send 1 records (since last report)
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
>
> org.apache.beam.sdk.io.kafka.KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo(FnApiDoFnRunner.java:1745)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.access$2700(FnApiDoFnRunner.java:142)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp(FnApiDoFnRunner.java:2263)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$ProcessBundleContextBase.output(FnApiDoFnRunner.java:2432)
>
> org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:78)
>
> org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:142)
>
> org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
> Source)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
>
> org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:172)
>
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver2.awaitCompletion(BeamFnDataInboundObserver2.java:126)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:467)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:151)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:116)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since
> last report)
>
> org.apache.beam.sdk.io.kafka.KafkaWriter.checkForFailures(KafkaWriter.java:133)
>
> org.apache.beam.sdk.io.kafka.KafkaWriter.processElement(KafkaWriter.java:58)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Topic
> beam-runnerv2 not present in metadata after 60000 ms.
> {noformat}
> run_rc_validation.sh fails with errors like:
> {noformat}
> * Please wait for at least 10 mins to let Dataflow job be launched and
> results get populated.
> * Sleeping for 10 mins
> * How to verify results:
> * 1. Goto your Dataflow job console and check whether there is any error.
> * 2. Check whether valentyn_python_validations_1229_10742.xlang_kafka_taxi
> has data, retrieving BigQuery data as below:
> BigQuery error in head operation: Not found: Table
> google.com:clouddfe:valentyn_python_validations_1229_10742.xlang_kafka_taxi
> Couldn't find expected output. Please confirm the output by visiting the
> console manually.
> {noformat}
> I suspect something is misconfigured but couldn't figure out what.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
