Hi Beam dev community,
I'm currently adding Beam python SDK support for my rust based portable beam
runner, initially it was designed around to execute java pipelines but, it
should be able to support majority of python pipeline proto translation logic,
because of portability.
However, I've run into a challenge, many Python pipeline pcollections are
encoded using the pickled coder (beam:coder:pickled_python:v1). I understand
this is a SDK specific (non-portable) coder and the runner cannot deserialize
it directly.
how should a runner handle a KV/ tuple encoded this way when performing GBK
For example, KV coder:
ref_Coder_TupleCoder_10": Coder {
spec: Some(
FunctionSpec {
urn: "beam:coder:kv:v1",
payload: [],
},
),
component_coder_ids: [
"ref_Coder_DeterministicFastPrimitivesCoderV2_v2_69_9",
"ref_Coder_IterableCoder_11",
],
},
Both its key and value are pickled coded:
"ref_Coder_DeterministicFastPrimitivesCoderV2_v2_69_9": Coder {
spec: Some(
FunctionSpec {
urn: "beam:coder:pickled_python:v1",
"ref_Coder_IterableCoder_11": Coder {
spec: Some(
FunctionSpec {
urn: "beam:coder:iterable:v1",
payload: [],
},
),
component_coder_ids: [
"ref_Coder_FastPrimitivesCoder_6",
],
},
"ref_Coder_FastPrimitivesCoder_6": Coder {
spec: Some(
FunctionSpec {
urn: "beam:coder:pickled_python:v1",
Whereas java pipeline proto would have beam primitive coder as its components,
making it straightforward to deserialize and group, how are portable runners
expected to handle raw pickle kv bytes
Thanks,
Ganesh.
