Valentyn Tymofieiev created BEAM-13579:
------------------------------------------
Summary: Cannot run python_xlang_kafka_taxi_dataflow validation
script on 2.35.0
Key: BEAM-13579
URL: https://issues.apache.org/jira/browse/BEAM-13579
Project: Beam
Issue Type: Bug
Components: release
Reporter: Valentyn Tymofieiev
Dataflow job gets errors like:
{noformat}
Error message from worker: generic::unknown:
org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter :
failed to send 1 records (since last report)
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
org.apache.beam.sdk.io.kafka.KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
Source)
org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo(FnApiDoFnRunner.java:1745)
org.apache.beam.fn.harness.FnApiDoFnRunner.access$2700(FnApiDoFnRunner.java:142)
org.apache.beam.fn.harness.FnApiDoFnRunner$NonWindowObservingProcessBundleContext.outputWithTimestamp(FnApiDoFnRunner.java:2263)
org.apache.beam.fn.harness.FnApiDoFnRunner$ProcessBundleContextBase.output(FnApiDoFnRunner.java:2432)
org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:78)
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:142)
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
Source)
org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:758)
org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:257)
org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:209)
org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:172)
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver2.awaitCompletion(BeamFnDataInboundObserver2.java:126)
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:467)
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:151)
org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:116)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since
last report)
org.apache.beam.sdk.io.kafka.KafkaWriter.checkForFailures(KafkaWriter.java:133)
org.apache.beam.sdk.io.kafka.KafkaWriter.processElement(KafkaWriter.java:58)
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic beam-runnerv2
not present in metadata after 60000 ms.
{noformat}
run_rc_validation.sh fails with errors like:
{noformat}
* Please wait for at least 10 mins to let Dataflow job be launched and results
get populated.
* Sleeping for 10 mins
* How to verify results:
* 1. Goto your Dataflow job console and check whether there is any error.
* 2. Check whether valentyn_python_validations_1229_10742.xlang_kafka_taxi has
data, retrieving BigQuery data as below:
BigQuery error in head operation: Not found: Table
google.com:clouddfe:valentyn_python_validations_1229_10742.xlang_kafka_taxi
Couldn't find expected output. Please confirm the output by visiting the
console manually.
{noformat}
I suspect something is misconfigured but couldn't figure out what.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
