Hey all, I'm not sure if I've missed something in the docs, but I'm having a bit of trouble with a streaming SQL job that starts w/ raw SQL queries and then transitions to a more traditional streaming job. I'm on Flink 1.10 using the Blink planner, running locally with no checkpointing.
The job looks roughly like: CSV 1 --> CSV 2 --> SQL Query to Join --> toRetractStream --> keyed time window w/ process func & custom trigger --> some other ops CSV 3 --> When I remove the windowing directly after the `toRetractStream`, the records make it to the "some other ops" stage, but with the windowing, those operations are sometimes not sent any data. I can also get data sent to the downstream operators by putting in a no-op map before the window and placing some breakpoints in there to manually slow down processing. The logs don't seem to indicate anything went wrong and generally look like: 4819 [Source: Custom File source (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4) switched from RUNNING to FINISHED.\4819 [Source: Custom File source (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4). 4819 [Source: Custom File source (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4) [FINISHED] 4820 [flink-akka.actor.default-dispatcher-5] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task Source: Custom File source (1/1) 3578629787c777320d9ab030c004abd4. ... 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, ProcessWindowFunction$1) (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f) switched from RUNNING to FINISHED. 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, ProcessWindowFunction$1) (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f). 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, ProcessWindowFunction$1) (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f) [FINISHED] ... rest of the shutdown ... Program execution finished Job with JobID 889b161e432c0e69a8d760bbed205d5d has finished. Job Runtime: 783 ms Is there something I'm missing in my setup? Could it be my custom window trigger? Bug? I'm stumped. Thanks, Austin
