I was able to get finer grained logs showing. I switched from -Dlog4j.configuration to -Dlog4j.configurationFile and it worked. With my larger test case, I was hitting a silent log4j error. When I created a small test case to just test logging, I received a log4j error.
Here is a tar <https://drive.google.com/file/d/1b6vJR_hfaRZwA28jKNlUBxDso7YiTIbk/view?usp=sharing> with the info logs for: - (test-nojoin.log) this one works as expected - (test-join.log) this does not work as expected I don't see an obvious issue just by scanning the logs. I'll take a deeper in 9 hours. On Wed, Oct 7, 2020 at 8:28 PM Dan Hill <quietgol...@gmail.com> wrote: > Switching to junit4 did not help. > > If I make a request to the url returned from > MiniClusterWithClientResource.flinkCluster.getClusterClient().getWebInterfaceURL(), > I get > {"errors":["Not found."]}. I'm not sure if this is intentional. > > > > > On Tue, Oct 6, 2020 at 4:16 PM Dan Hill <quietgol...@gmail.com> wrote: > >> @Aljoscha - Thanks! That setup lets fixing the hacky absolute path >> reference. However, the actual log calls are not printing to the console. >> Only errors appear in my terminal window and the test logs. Maybe console >> logger does not work for this junit setup. I'll see if the file version >> works. >> >> On Tue, Oct 6, 2020 at 4:08 PM Austin Cawley-Edwards < >> austin.caw...@gmail.com> wrote: >> >>> What Aljoscha suggested is what works for us! >>> >>> On Tue, Oct 6, 2020 at 6:58 PM Aljoscha Krettek <aljos...@apache.org> >>> wrote: >>> >>>> Hi Dan, >>>> >>>> to make the log properties file work this should do it: assuming the >>>> log4j.properties is in //src/main/resources. You will need a >>>> BUILD.bazel >>>> in that directory that has only the line >>>> "exports_files(["log4j.properties"]). Then you can reference it in your >>>> test via "resources = ["//src/main/resources:log4j.properties"],". Of >>>> course you also need to have the right log4j deps (or slf4j if you're >>>> using that) >>>> >>>> Hope that helps! >>>> >>>> Aljoscha >>>> >>>> On 07.10.20 00:41, Dan Hill wrote: >>>> > I'm trying to use Table API for my job. I'll soon try to get a test >>>> > working for my stream job. >>>> > - I'll parameterize so I can have different sources and sink for >>>> tests. >>>> > How should I mock out a Kafka source? For my test, I was planning on >>>> > changing the input to be from a temp file (instead of Kafka). >>>> > - What's a good way of forcing a watermark using the Table API? >>>> > >>>> > >>>> > On Tue, Oct 6, 2020 at 3:35 PM Dan Hill <quietgol...@gmail.com> >>>> wrote: >>>> > >>>> >> Thanks! >>>> >> >>>> >> Great to know. I copied this junit5-jupiter-starter-bazel >>>> >> < >>>> https://github.com/junit-team/junit5-samples/tree/main/junit5-jupiter-starter-bazel> >>>> rule >>>> >> into my repository (I don't think junit5 is supported directly with >>>> >> java_test yet). I tried a few ways of bundling `log4j.properties` >>>> into the >>>> >> jar and didn't get them to work. My current iteration hacks the >>>> >> log4j.properties file as an absolute path. My failed attempts would >>>> spit >>>> >> an error saying log4j.properties file was not found. This route >>>> finds it >>>> >> but the log properties are not used for the java logger. >>>> >> >>>> >> Are there a better set of rules to use for junit5? >>>> >> >>>> >> # build rule >>>> >> java_junit5_test( >>>> >> name = "tests", >>>> >> srcs = glob(["*.java"]), >>>> >> test_package = "ai.promoted.logprocessor.batch", >>>> >> deps = [...], >>>> >> jvm_flags = >>>> >> >>>> ["-Dlog4j.configuration=file:///Users/danhill/code/src/ai/promoted/logprocessor/batch/log4j.properties"], >>>> >> ) >>>> >> >>>> >> # log4j.properties >>>> >> status = error >>>> >> name = Log4j2PropertiesConfig >>>> >> appenders = console >>>> >> appender.console.type = Console >>>> >> appender.console.name = LogToConsole >>>> >> appender.console.layout.type = PatternLayout >>>> >> appender.console.layout.pattern = %d [%t] %-5p %c - %m%n >>>> >> rootLogger.level = info >>>> >> rootLogger.appenderRefs = stdout >>>> >> rootLogger.appenderRef.stdout.ref = LogToConsole >>>> >> >>>> >> On Tue, Oct 6, 2020 at 3:34 PM Austin Cawley-Edwards < >>>> >> austin.caw...@gmail.com> wrote: >>>> >> >>>> >>> Oops, this is actually the JOIN issue thread [1]. Guess I should >>>> revise >>>> >>> my previous "haven't had issues" statement hah. Sorry for the spam! >>>> >>> >>>> >>> [1]: >>>> >>> >>>> apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Streaming-SQL-Job-Switches-to-FINISHED-before-all-records-processed-td38382.html >>>> >>> >>>> >>> On Tue, Oct 6, 2020 at 6:32 PM Austin Cawley-Edwards < >>>> >>> austin.caw...@gmail.com> wrote: >>>> >>> >>>> >>>> Unless it's related to this issue[1], which was w/ my JOIN and time >>>> >>>> characteristics, though not sure that applies for batch. >>>> >>>> >>>> >>>> Best, >>>> >>>> Austin >>>> >>>> >>>> >>>> [1]: >>>> >>>> >>>> apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-Streaming-Join-Creates-Duplicates-td37764.html >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Oct 6, 2020 at 6:20 PM Austin Cawley-Edwards < >>>> >>>> austin.caw...@gmail.com> wrote: >>>> >>>> >>>> >>>>> Hey Dan, >>>> >>>>> >>>> >>>>> We use Junit5 and Bazel to run Flink SQL tests on a mini cluster >>>> and >>>> >>>>> haven’t had issues, though we’re only testing on streaming jobs. >>>> >>>>> >>>> >>>>> Happy to help setting up logging with that if you’d like. >>>> >>>>> >>>> >>>>> Best, >>>> >>>>> Austin >>>> >>>>> >>>> >>>>> On Tue, Oct 6, 2020 at 6:02 PM Dan Hill <quietgol...@gmail.com> >>>> wrote: >>>> >>>>> >>>> >>>>>> I don't think any of the gotchas apply to me (at the bottom of >>>> this >>>> >>>>>> link). >>>> >>>>>> >>>> >>>>>> >>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#junit-rule-miniclusterwithclientresource >>>> >>>>>> >>>> >>>>>> I'm assuming for a batch job that I don't have to do anything >>>> for: >>>> >>>>>> "You can implement a custom parallel source function for emitting >>>> >>>>>> watermarks if your job uses event time timers." >>>> >>>>>> >>>> >>>>>> On Tue, Oct 6, 2020 at 2:42 PM Dan Hill <quietgol...@gmail.com> >>>> wrote: >>>> >>>>>> >>>> >>>>>>> I've tried to enable additional logging for a few hours today. >>>> I >>>> >>>>>>> think something with junit5 is swallowing the logs. I'm using >>>> Bazel and >>>> >>>>>>> junit5. I setup MiniClusterResourceConfiguration using a custom >>>> >>>>>>> extension. Are there any known issues with Flink and junit5? >>>> I can try >>>> >>>>>>> switching to junit4. >>>> >>>>>>> >>>> >>>>>>> When I've binary searched this issue, this failure happens if my >>>> >>>>>>> query in step 3 has a join it. If I remove the join, I can >>>> remove step 4 >>>> >>>>>>> and the code still works. I've renamed a bunch of my tables >>>> too and the >>>> >>>>>>> problem still exists. >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> On Tue, Oct 6, 2020, 00:42 Aljoscha Krettek < >>>> aljos...@apache.org> >>>> >>>>>>> wrote: >>>> >>>>>>> >>>> >>>>>>>> Hi Dan, >>>> >>>>>>>> >>>> >>>>>>>> there were some bugs and quirks in the MiniCluster that we >>>> recently >>>> >>>>>>>> fixed: >>>> >>>>>>>> >>>> >>>>>>>> - https://issues.apache.org/jira/browse/FLINK-19123 >>>> >>>>>>>> - https://issues.apache.org/jira/browse/FLINK-19264 >>>> >>>>>>>> >>>> >>>>>>>> But I think they are probably unrelated to your case. Could you >>>> >>>>>>>> enable >>>> >>>>>>>> logging and see from the logs whether the 2) and 3) jobs >>>> execute >>>> >>>>>>>> correctly on the MiniCluster? >>>> >>>>>>>> >>>> >>>>>>>> Best, >>>> >>>>>>>> Aljoscha >>>> >>>>>>>> >>>> >>>>>>>> On 06.10.20 08:08, Dan Hill wrote: >>>> >>>>>>>>> I'm writing a test for a batch job using >>>> >>>>>>>> MiniClusterResourceConfiguration. >>>> >>>>>>>>> >>>> >>>>>>>>> Here's a simple description of my working test case: >>>> >>>>>>>>> 1) I use TableEnvironment.executeSql(...) to create a source >>>> and >>>> >>>>>>>> sink table >>>> >>>>>>>>> using tmp filesystem directory. >>>> >>>>>>>>> 2) I use executeSql to insert some test data into the source >>>> tabel. >>>> >>>>>>>>> 3) I use executeSql to select from source and insert into >>>> sink. >>>> >>>>>>>>> 4) I use executeSql from the same source to a different sink. >>>> >>>>>>>>> >>>> >>>>>>>>> When I do these steps, it works. If I remove step 4, no data >>>> gets >>>> >>>>>>>> written >>>> >>>>>>>>> to the sink. My actual code is more complex than this (has >>>> create >>>> >>>>>>>> view, >>>> >>>>>>>>> join and more tables). This is a simplified description but >>>> >>>>>>>> highlights the >>>> >>>>>>>>> weird error. >>>> >>>>>>>>> >>>> >>>>>>>>> Has anyone hit issues like this? I'm assuming I have a small >>>> code >>>> >>>>>>>> bug in >>>> >>>>>>>>> my queries that's causing issues. These queries appear to >>>> work in >>>> >>>>>>>>> production so I'm confused. Are there ways of viewing failed >>>> jobs >>>> >>>>>>>> or >>>> >>>>>>>>> queries with MiniClusterResourceConfiguration? >>>> >>>>>>>>> >>>> >>>>>>>>> Thanks! >>>> >>>>>>>>> - Dan >>>> >>>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> > >>>> >>>>
