Hi All - I am facing an issue while using *side-input*.
*What am I doing:*
>From my main program, I am calling a custom PTransform with a
PCollectionView as parameter. Inside custom PTransform, I am passing the
PCollectionView as a side-input to a DoFn.
*Issue:*
When I run the pipeline, I am expecting the log statement inside my DoFn's
processElement to get executed but it is not getting logged. If I remove
the side-input to my DoFn then the log is getting printed. I am suspecting
whether it could be related to windowing/execution order or my side-input
somehow being empty. Appreciate if you can clarify on what is going wrong
here.
*Code Structure:*
*Main Program:* PCollectionTuple tuple = input.apply(new FirstTx());
// Get two tuple tags from first transformation
PCollection1 = tuple.get(tag1).setCoder(...);
PCollection2 = tuple.get(tag2).setCoder(...);
// Converting PCollection1 to PCollectionView to use as a side-input
// Note: I need to introduce a global window here as my source is
unbounded and when we use View.asList() it does GroupByKey internally
which inturn demands a window
PView = PCollection1.apply(Window.<KV<String, CustomObject>>into(new
GlobalWindows()) // Everything into global window.
.triggering(Repeatedly.forever(DefaultTrigger.of()))
.discardingFiredPanes()).apply(Values.create()).apply(View.asList());
// Pass PCollectionView to SecondTx as a param
PCollection3 = PCollection2.apply(new SecondTx(PView));
*SecondTx:*
Inside my SecondTx, I am getting the PView from constructor (this.PView =
PView) and calling a DoFn
public PCollection<CustomObject> expand(PCollection <KV <String, KV
<String, CustomObject>>> input) {
input.apply(ParDo.of(new UpdateFn()).withSideInput("SideInput", PView));
...
}
// DoFn
class UpdateFn extends DoFn<Map<String, Map<String, Map<String, String>>>,
CustomObject> {
@ProcessElement
public void processElement(@Element Map<String, Map<String, Map<String,
String>>> input, OutputReceiver<CustomObject> out) {
* Log.of("UpdateFn " + input);*
out.output(new CustomObject());
}
}
--
Thanks,
Praveen K Viswanathan