Oops, this is actually the JOIN issue thread [1]. Guess I should revise my previous "haven't had issues" statement hah. Sorry for the spam!
[1]: apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Streaming-SQL-Job-Switches-to-FINISHED-before-all-records-processed-td38382.html On Tue, Oct 6, 2020 at 6:32 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Unless it's related to this issue[1], which was w/ my JOIN and time > characteristics, though not sure that applies for batch. > > Best, > Austin > > [1]: > apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-Streaming-Join-Creates-Duplicates-td37764.html > > > On Tue, Oct 6, 2020 at 6:20 PM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hey Dan, >> >> We use Junit5 and Bazel to run Flink SQL tests on a mini cluster and >> haven’t had issues, though we’re only testing on streaming jobs. >> >> Happy to help setting up logging with that if you’d like. >> >> Best, >> Austin >> >> On Tue, Oct 6, 2020 at 6:02 PM Dan Hill <quietgol...@gmail.com> wrote: >> >>> I don't think any of the gotchas apply to me (at the bottom of this >>> link). >>> >>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#junit-rule-miniclusterwithclientresource >>> >>> I'm assuming for a batch job that I don't have to do anything for: "You >>> can implement a custom parallel source function for emitting watermarks if >>> your job uses event time timers." >>> >>> On Tue, Oct 6, 2020 at 2:42 PM Dan Hill <quietgol...@gmail.com> wrote: >>> >>>> I've tried to enable additional logging for a few hours today. I think >>>> something with junit5 is swallowing the logs. I'm using Bazel and junit5. >>>> I setup MiniClusterResourceConfiguration using a custom extension. Are >>>> there any known issues with Flink and junit5? I can try switching to >>>> junit4. >>>> >>>> When I've binary searched this issue, this failure happens if my query >>>> in step 3 has a join it. If I remove the join, I can remove step 4 and the >>>> code still works. I've renamed a bunch of my tables too and the problem >>>> still exists. >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Oct 6, 2020, 00:42 Aljoscha Krettek <aljos...@apache.org> >>>> wrote: >>>> >>>>> Hi Dan, >>>>> >>>>> there were some bugs and quirks in the MiniCluster that we recently >>>>> fixed: >>>>> >>>>> - https://issues.apache.org/jira/browse/FLINK-19123 >>>>> - https://issues.apache.org/jira/browse/FLINK-19264 >>>>> >>>>> But I think they are probably unrelated to your case. Could you enable >>>>> logging and see from the logs whether the 2) and 3) jobs execute >>>>> correctly on the MiniCluster? >>>>> >>>>> Best, >>>>> Aljoscha >>>>> >>>>> On 06.10.20 08:08, Dan Hill wrote: >>>>> > I'm writing a test for a batch job using >>>>> MiniClusterResourceConfiguration. >>>>> > >>>>> > Here's a simple description of my working test case: >>>>> > 1) I use TableEnvironment.executeSql(...) to create a source and >>>>> sink table >>>>> > using tmp filesystem directory. >>>>> > 2) I use executeSql to insert some test data into the source tabel. >>>>> > 3) I use executeSql to select from source and insert into sink. >>>>> > 4) I use executeSql from the same source to a different sink. >>>>> > >>>>> > When I do these steps, it works. If I remove step 4, no data gets >>>>> written >>>>> > to the sink. My actual code is more complex than this (has create >>>>> view, >>>>> > join and more tables). This is a simplified description but >>>>> highlights the >>>>> > weird error. >>>>> > >>>>> > Has anyone hit issues like this? I'm assuming I have a small code >>>>> bug in >>>>> > my queries that's causing issues. These queries appear to work in >>>>> > production so I'm confused. Are there ways of viewing failed jobs or >>>>> > queries with MiniClusterResourceConfiguration? >>>>> > >>>>> > Thanks! >>>>> > - Dan >>>>> > >>>>> >>>>>
