Unless it's related to this issue[1], which was w/ my JOIN and time characteristics, though not sure that applies for batch.
Best, Austin [1]: apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-Streaming-Join-Creates-Duplicates-td37764.html On Tue, Oct 6, 2020 at 6:20 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Dan, > > We use Junit5 and Bazel to run Flink SQL tests on a mini cluster and > haven’t had issues, though we’re only testing on streaming jobs. > > Happy to help setting up logging with that if you’d like. > > Best, > Austin > > On Tue, Oct 6, 2020 at 6:02 PM Dan Hill <quietgol...@gmail.com> wrote: > >> I don't think any of the gotchas apply to me (at the bottom of this link). >> >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#junit-rule-miniclusterwithclientresource >> >> I'm assuming for a batch job that I don't have to do anything for: "You >> can implement a custom parallel source function for emitting watermarks if >> your job uses event time timers." >> >> On Tue, Oct 6, 2020 at 2:42 PM Dan Hill <quietgol...@gmail.com> wrote: >> >>> I've tried to enable additional logging for a few hours today. I think >>> something with junit5 is swallowing the logs. I'm using Bazel and junit5. >>> I setup MiniClusterResourceConfiguration using a custom extension. Are >>> there any known issues with Flink and junit5? I can try switching to >>> junit4. >>> >>> When I've binary searched this issue, this failure happens if my query >>> in step 3 has a join it. If I remove the join, I can remove step 4 and the >>> code still works. I've renamed a bunch of my tables too and the problem >>> still exists. >>> >>> >>> >>> >>> >>> On Tue, Oct 6, 2020, 00:42 Aljoscha Krettek <aljos...@apache.org> wrote: >>> >>>> Hi Dan, >>>> >>>> there were some bugs and quirks in the MiniCluster that we recently >>>> fixed: >>>> >>>> - https://issues.apache.org/jira/browse/FLINK-19123 >>>> - https://issues.apache.org/jira/browse/FLINK-19264 >>>> >>>> But I think they are probably unrelated to your case. Could you enable >>>> logging and see from the logs whether the 2) and 3) jobs execute >>>> correctly on the MiniCluster? >>>> >>>> Best, >>>> Aljoscha >>>> >>>> On 06.10.20 08:08, Dan Hill wrote: >>>> > I'm writing a test for a batch job using >>>> MiniClusterResourceConfiguration. >>>> > >>>> > Here's a simple description of my working test case: >>>> > 1) I use TableEnvironment.executeSql(...) to create a source and sink >>>> table >>>> > using tmp filesystem directory. >>>> > 2) I use executeSql to insert some test data into the source tabel. >>>> > 3) I use executeSql to select from source and insert into sink. >>>> > 4) I use executeSql from the same source to a different sink. >>>> > >>>> > When I do these steps, it works. If I remove step 4, no data gets >>>> written >>>> > to the sink. My actual code is more complex than this (has create >>>> view, >>>> > join and more tables). This is a simplified description but >>>> highlights the >>>> > weird error. >>>> > >>>> > Has anyone hit issues like this? I'm assuming I have a small code >>>> bug in >>>> > my queries that's causing issues. These queries appear to work in >>>> > production so I'm confused. Are there ways of viewing failed jobs or >>>> > queries with MiniClusterResourceConfiguration? >>>> > >>>> > Thanks! >>>> > - Dan >>>> > >>>> >>>>
