[
https://issues.apache.org/jira/browse/BEAM-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853197#comment-16853197
]
Alex Amato edited comment on BEAM-7462 at 5/31/19 4:48 PM:
-----------------------------------------------------------
Here is a dump of the bundle descriptor, showing the code and pcollectionId
which triggers the issue.
[^bundle_descriptor_dump.txt][^bundle_descriptor_dump.txt]
At first I confused this with generatedPCollection-23, which is real. But
generatedPCollection-203 does not appear in the bundle descriptor. Is it
produced elsewhere? Oddly this error comes from a pcollection-203 ID which does
not exist in the descriptor.
And here is the error that occurred.
sample with pCollectionId: *generatedPcollection-203* pColl.getName():
generatedPcollection-203 Coder:
*org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder*
{
insertId: "2018788043244039965:565032:0:164342"
jsonPayload:
{ exception: "java.lang.ClassCastException: [B cannot be cast to
org.apache.beam.sdk.util.WindowedValue at
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:480)
at
org.apache.beam.fn.harness.data.ElementCountFnDataReceiver.accept(ElementCountFnDataReceiver.java:113)
at
org.apache.beam.fn.harness.data.ElementCountFnDataReceiver.accept(ElementCountFnDataReceiver.java:45)
at
org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:108)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:332)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:160)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) " job:
"2019-05-30_14_57_35-14659722017552460993" logger:
"org.apache.beam.fn.harness.control.BeamFnControlClient" message: "Exception
while trying to handle InstructionRequest -212" portability_worker_id: "1"
thread: "17" worker: "testpipeline-jenkins-0530-05301457-7v4d-harness-bbc4" }
labels:
{…}
logName: "projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fworker"
receiveTimestamp: "2019-05-30T22:01:04.911873820Z"
resource: \{…}
severity: "ERROR"
timestamp: "2019-05-30T22:00:55.189Z"
}
was (Author: [email protected]):
Here is a dump of the bundle descriptor, showing the code and pcollectionId
which trigger the issue.
[^bundle_descriptor_dump.txt][^bundle_descriptor_dump.txt]
} transforms {
key: "generatedPtransform-26" value {
spec {
*urn: "beam:transform:pardo:v1" payload:*
"\n\356\004\n\327\004\n\025beam:dofn:javasdk:0.1\032\275\004\202SNAPPY\000\000\000\000\001\000\000\000\001\000\000\002)\347\006\360a\254\355\000\005sr\000?org.apache.beam.sdk.util.AutoValue_DoFnWithExecutionInformationo_\336e?\346.
{
\002\000\003L\000\004doFnt\000%Lorg/a\005U\000/\001U</sdk/transforms/\001Q@;L\000\rmainOutputTag^8\000dvalues/TupleTag;L\000\021schemaI\031\206\bt\0006\216t\000\000S>8\000\024;xr\0005o^\002\001\001\247^\370\000D9\003c\264\241\274\276\220\002\000\000xpsr\000CoJF\000\031\363\360<.Reshuffle$ViaRandomKey$AssignShardFn\200\363Y\322\244\270\274\224\002\000\001I\000\005shardxr\000#ovZ\0000DoFn\254D#\006\316\223\275\235\002\001\216\020\000\000\000\000sZ8\000)M\000.1M\330\263\030yf[\300z\265\002\000\002Z\000\tgeneratedL\000\002idt\000\022Ljava/lang/String;xp\001t\000BN\354\000\rZ\250PCollection.<init>:402#b70c45c110743c2bsr\000>NF\000=2\004Au.\200\002\000S>\273\001\210\3425\313\323\267`\232\254\002\000\001L\000\021elementConverterst\000\020L\005\2764util/List;xr\0004zt\000!LBj\000\034\257\325QB\327
}
\001W%]\fsr\000\037\001Xi$\031\355\030s$Empty\001i0z\270\027\264<\247\236\336\002\000\000xp\022\022beam:env:docker:v1"
} inputs
{ key: "generatedInput-27" value: "generatedPcollection-19" }
outputs
{ *key: "org.apache.beam.sdk.values.PCollection.<init>:402#b70c45c110743c2b"
value: "*generatedPcollection-23*"* }
} transforms {
key: "-14" value {
spec
{ *urn: "beam:sink:runner:0.1" payload: "\n\021\022\017localhost:12371"* }
inputs
{ *key: "-13" value: "*generatedPcollection-23*"* }
} pcollections {
key: "*generatedPcollection-23*" value {
coder_id: "*generatedCoder-22*" is_bounded: BOUNDED windowing_strategy_id:
"fakeWindowingStrategy-17"
}
}
} coders {
*key: "generatedCoder-22"* value {
spec {
*urn: "beam:coder:windowed_value:v1"*
} component_coder_ids: "KvCoder14" component_coder_ids: "Coder1"
}
sample with pCollectionId: *generatedPcollection-203* pColl.getName():
generatedPcollection-203 Coder:
*org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder*
{
insertId: "2018788043244039965:565032:0:164342"
jsonPayload:
{ exception: "java.lang.ClassCastException: [B cannot be cast to
org.apache.beam.sdk.util.WindowedValue at
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:480)
at
org.apache.beam.fn.harness.data.ElementCountFnDataReceiver.accept(ElementCountFnDataReceiver.java:113)
at
org.apache.beam.fn.harness.data.ElementCountFnDataReceiver.accept(ElementCountFnDataReceiver.java:45)
at
org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:108)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:332)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:160)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) " job:
"2019-05-30_14_57_35-14659722017552460993" logger:
"org.apache.beam.fn.harness.control.BeamFnControlClient" message: "Exception
while trying to handle InstructionRequest -212" portability_worker_id: "1"
thread: "17" worker: "testpipeline-jenkins-0530-05301457-7v4d-harness-bbc4" }
labels:
{…}
logName: "projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fworker"
receiveTimestamp: "2019-05-30T22:01:04.911873820Z"
resource: \{…}
severity: "ERROR"
timestamp: "2019-05-30T22:00:55.189Z"
}
> Add Sampled Byte Count Metric to the Java SDK
> ---------------------------------------------
>
> Key: BEAM-7462
> URL: https://issues.apache.org/jira/browse/BEAM-7462
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution
> Reporter: Alex Amato
> Priority: Major
> Attachments: bundle_descriptor_dump.txt
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
