Re: Extremely large job serialization produced by union operator

2018-03-14 Thread Fabian Hueske
Can you share the operator plan (StreamExecutionEnvironment.getExecutionPlan()) for both cases? Thanks, Fabian 2018-03-14 9:08 GMT+01:00 杨力 : > I understand complex SQL queries would be translated into large DAGs. > However, the submission succeeds in my case if I don't use union operator. > It

Re: Extremely large job serialization produced by union operator

2018-03-14 Thread 杨力
I understand complex SQL queries would be translated into large DAGs. However, the submission succeeds in my case if I don't use union operator. It might be a potential bug related to it. For example, following code submisses successfully with the default limitations of akka.framesize. val sqls: S

Re: Extremely large job serialization produced by union operator

2018-03-13 Thread Fabian Hueske
Hi Bill, The size of the program depends on the number and complexity SQL queries that you are submitting. Each query might be translated into a sequence of multiple operators. Each operator has a string with generated code that will be compiled on the worker nodes. The size of the code depends on

Re: Extremely large job serialization produced by union operator

2018-03-09 Thread 杨力
Thank you for your response. It occurs both in a standalone cluster anda a yarn-cluster. I am trying to remove business code and reproduce it with a minimal demo. On Sat, Mar 10, 2018 at 2:27 AM Piotr Nowojski wrote: > Hi, > > Could you provide more details about your queries and setup? Logs cou

Re: Extremely large job serialization produced by union operator

2018-03-09 Thread Piotr Nowojski
Hi, Could you provide more details about your queries and setup? Logs could be helpful as well. Piotrek > On 9 Mar 2018, at 11:00, 杨力 wrote: > > I wrote a flink-sql app with following topography. > > KafkaJsonTableSource -> SQL -> toAppendStream -> Map -> JDBCAppendTableSink > KafkaJsonTable