Hey all, Thanks for your patience. I've got a small repo that reproduces the issue here: https://github.com/austince/flink-1.10-sql-windowing-error
Not sure what I'm doing wrong but it feels silly. Thanks so much! Austin On Tue, Sep 29, 2020 at 3:48 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Till, > > Thanks for the reply -- I'll try to see if I can reproduce this in a small > repo and share it with you. > > Best, > Austin > > On Tue, Sep 29, 2020 at 3:58 AM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Austin, >> >> could you share with us the exact job you are running (including the >> custom window trigger)? This would help us to better understand your >> problem. >> >> I am also pulling in Klou and Timo who might help with the windowing >> logic and the Table to DataStream conversion. >> >> Cheers, >> Till >> >> On Mon, Sep 28, 2020 at 11:49 PM Austin Cawley-Edwards < >> austin.caw...@gmail.com> wrote: >> >>> Hey all, >>> >>> I'm not sure if I've missed something in the docs, but I'm having a bit >>> of trouble with a streaming SQL job that starts w/ raw SQL queries and then >>> transitions to a more traditional streaming job. I'm on Flink 1.10 using >>> the Blink planner, running locally with no checkpointing. >>> >>> The job looks roughly like: >>> >>> CSV 1 --> >>> CSV 2 --> SQL Query to Join --> toRetractStream --> keyed time window >>> w/ process func & custom trigger --> some other ops >>> CSV 3 --> >>> >>> >>> When I remove the windowing directly after the `toRetractStream`, the >>> records make it to the "some other ops" stage, but with the windowing, >>> those operations are sometimes not sent any data. I can also get data sent >>> to the downstream operators by putting in a no-op map before the window and >>> placing some breakpoints in there to manually slow down processing. >>> >>> >>> The logs don't seem to indicate anything went wrong and generally look >>> like: >>> >>> 4819 [Source: Custom File source (1/1)] INFO >>> org.apache.flink.runtime.taskmanager.Task - Source: Custom File source >>> (1/1) (3578629787c777320d9ab030c004abd4) switched from RUNNING to >>> FINISHED.\4819 [Source: Custom File source (1/1)] INFO >>> org.apache.flink.runtime.taskmanager.Task - Freeing task resources for >>> Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4). >>> 4819 [Source: Custom File source (1/1)] INFO >>> org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem >>> streams are closed for task Source: Custom File source (1/1) >>> (3578629787c777320d9ab030c004abd4) [FINISHED] >>> 4820 [flink-akka.actor.default-dispatcher-5] INFO >>> org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task >>> and sending final execution state FINISHED to JobManager for task Source: >>> Custom File source (1/1) 3578629787c777320d9ab030c004abd4. >>> ... >>> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>> ProcessWindowFunction$1) (1/1)] INFO >>> org.apache.flink.runtime.taskmanager.Task - >>> Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>> ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f) switched >>> from RUNNING to FINISHED. >>> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>> ProcessWindowFunction$1) (1/1)] INFO >>> org.apache.flink.runtime.taskmanager.Task - Freeing task resources for >>> Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>> ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f). >>> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>> ProcessWindowFunction$1) (1/1)] INFO >>> org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem >>> streams are closed for task Window(TumblingProcessingTimeWindows(60000), >>> TimedCountTrigger, ProcessWindowFunction$1) (1/1) >>> (907acf9bfa2f4a9bbd13997b8b34d91f) [FINISHED] >>> ... >>> rest of the shutdown >>> ... >>> Program execution finished >>> Job with JobID 889b161e432c0e69a8d760bbed205d5d has finished. >>> Job Runtime: 783 ms >>> >>> >>> Is there something I'm missing in my setup? Could it be my custom window >>> trigger? Bug? I'm stumped. >>> >>> >>> Thanks, >>> Austin >>> >>> >>>
