[
https://issues.apache.org/jira/browse/BEAM-12021?focusedWorklogId=583008&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-583008
]
ASF GitHub Bot logged work on BEAM-12021:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Apr/21 23:26
Start Date: 14/Apr/21 23:26
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#14276:
URL: https://github.com/apache/beam/pull/14276#discussion_r613575491
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -2344,31 +2300,22 @@ static String
getContainerImageForJob(DataflowPipelineOptions options) {
}
static boolean useUnifiedWorker(DataflowPipelineOptions options) {
- return hasExperiment(options, "beam_fn_api")
- || hasExperiment(options, "use_runner_v2")
- || hasExperiment(options, "use_unified_worker");
+ return hasExperiment(options, "beam_fn_api");
Review comment:
Done. I suppose soon we simply delete this method.
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -2344,31 +2300,22 @@ static String
getContainerImageForJob(DataflowPipelineOptions options) {
}
static boolean useUnifiedWorker(DataflowPipelineOptions options) {
- return hasExperiment(options, "beam_fn_api")
- || hasExperiment(options, "use_runner_v2")
- || hasExperiment(options, "use_unified_worker");
+ return hasExperiment(options, "beam_fn_api");
}
static boolean useStreamingEngine(DataflowPipelineOptions options) {
return hasExperiment(options, GcpOptions.STREAMING_ENGINE_EXPERIMENT)
|| hasExperiment(options, GcpOptions.WINDMILL_SERVICE_EXPERIMENT);
}
- static void verifyDoFnSupported(
- DoFn<?, ?> fn, boolean streaming, boolean workerV2, boolean
streamingEngine) {
+ static void verifyDoFnSupported(DoFn<?, ?> fn, boolean streaming, boolean
streamingEngine) {
if (streaming && DoFnSignatures.requiresTimeSortedInput(fn)) {
throw new UnsupportedOperationException(
String.format(
"%s does not currently support @RequiresTimeSortedInput in
streaming mode.",
DataflowRunner.class.getSimpleName()));
}
if (DoFnSignatures.usesSetState(fn)) {
- if (workerV2) {
Review comment:
Done
##########
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
##########
@@ -1334,57 +1329,6 @@ public void testTransformTranslator() throws IOException
{
assertTrue(transform.translated);
}
- @Test
- public void testSdkHarnessConfiguration() throws IOException {
Review comment:
When I wrote this, I thought this was testing an unused case. Is that
not true? I have restored the test, to investigate if needed...
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -1005,7 +961,7 @@ public DataflowPipelineJob run(Pipeline pipeline) {
options.getStager().stageToFile(serializedProtoPipeline,
PIPELINE_FILE_NAME);
dataflowOptions.setPipelineUrl(stagedPipeline.getLocation());
// Now rewrite things to be as needed for v1 (mutates the pipeline)
- replaceTransforms(pipeline);
+ replaceV1Transforms(pipeline);
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 583008)
Time Spent: 8h 20m (was: 8h 10m)
> PubsubReadIT failures: "Cannot nackAll on persisting checkpoint"
> ----------------------------------------------------------------
>
> Key: BEAM-12021
> URL: https://issues.apache.org/jira/browse/BEAM-12021
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, test-failures
> Reporter: Tyson Hamilton
> Assignee: Kenneth Knowles
> Priority: P1
> Labels: currently-failing
> Fix For: Not applicable
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> *
> [org.apache.beam.sdk.io.gcp.pubsub.PubsubReadIT.testReadPubsubMessageId|https://ci-beam.apache.org/job/beam_PostCommit_Java/7332/testReport/junit/org.apache.beam.sdk.io.gcp.pubsub/PubsubReadIT/testReadPubsubMessageId_2/]
> *
> [org.apache.beam.sdk.io.gcp.pubsub.PubsubReadIT.testReadPublicData|https://ci-beam.apache.org/job/beam_PostCommit_Java/7332/testReport/junit/org.apache.beam.sdk.io.gcp.pubsub/PubsubReadIT/testReadPublicData_2/]
> Job:
> [https://console.cloud.google.com/dataflow/jobs/us-central1/2021-03-17_14_02_29-12611938587996322031?project=apache-beam-testing]
>
> Many worker errors (754) that look like:
> {code:java}
> 2021-03-17 20:47:29.000 PDTError message from worker: generic::unknown:
> java.lang.IllegalStateException: Cannot nackAll on persisting checkpoint
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:507)
>
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource$PubsubCheckpoint.nackAll(PubsubUnboundedSource.java:308)
>
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource$PubsubSource.createReader(PubsubUnboundedSource.java:1071)
>
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource$PubsubSource.createReader(PubsubUnboundedSource.java:1012)
>
> org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$UnboundedSourceAsSDFRestrictionTracker.getProgress(Read.java:963)
>
> org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DefaultGetSize.invokeGetSize(ByteBuddyDoFnInvokerFactory.java:426)
>
> org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$DoFnInvoker.invokeGetSize(Unknown
> Source)
> org.apache.beam.fn.harness.FnApiDoFnRunner.calculateRestrictionSize(FnApiDoFnRunner.java:1182)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.trySplitForElementAndRestriction(FnApiDoFnRunner.java:1608)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForWindowObservingSizedElementAndRestriction(FnApiDoFnRunner.java:1059)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner.access$1000(FnApiDoFnRunner.java:139)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$4.accept(FnApiDoFnRunner.java:638)
> org.apache.beam.fn.harness.FnApiDoFnRunner$4.accept(FnApiDoFnRunner.java:633)
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:246)
>
> org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:200)
>
> org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:220)
>
> org.apache.beam.sdk.fn.data.DecodingFnDataReceiver.accept(DecodingFnDataReceiver.java:43)
>
> org.apache.beam.sdk.fn.data.DecodingFnDataReceiver.accept(DecodingFnDataReceiver.java:25)
>
> org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:111)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:308)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748) passed through: ==>
> dist_proc/dax/workflow/worker/fnapi_service.cc:631
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
