Re: Ship compiled code with broadcast stream ?

2018-10-09 Thread Gyula Fóra
You should not try sending the compiled code anywhere but you can use it from within the processor. You can do the same thing with the jar, you compile your jar, store it on HDFS. Send the jar path to the processor which can download the jar and instantiate the rule. No need to resubmit the job.

Re: Ship compiled code with broadcast stream ?

2018-10-09 Thread Maxim Parkachov
Hi, This is certainly possible. What you can do is use a > BroadcastProcessFunction where you receive the rule code on the broadcast > side. > Yes, this part works, no problem. > You probably cannot send newly compiled objects this way but what you can > do is either send a reference to some

Re: Ship compiled code with broadcast stream ?

2018-10-09 Thread Gyula Fóra
Hi, This is certainly possible. What you can do is use a BroadcastProcessFunction where you receive the rule code on the broadcast side. You probably cannot send newly compiled objects this way but what you can do is either send a reference to some compiled jars and load them with the

Ship compiled code with broadcast stream ?

2018-10-09 Thread Maxim Parkachov
Hi everyone, I have a job with event stream and control stream delivering rules for event transformation. Rules are broadcasted and used in flatMat-like coProcessFunction. Rules are defined in custom JSON format. Amount of rules and complexity rises significantly with every new feature. What I