Just saw a similar problem for the first time with `
:beam-sdks-java-io-clickhouse:compileJava`; I got the errors below toward
the end of a full-project compile, and when I re-ran just that target,
compilation succeeded.
Intriguingly, extensions/sql and clickhouse are the only two modules that
use the compileJavacc task to generate code that then gets compileJava'd.
Something is up with it!
I looked at the debug log from my previous email a bit, but haven't found
anything.
```
> Task :beam-sdks-java-io-clickhouse:compileJava
…/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/TableSchema.java:228:
error: cannot find symbol
return new
org.apache.beam.sdk.io.clickhouse.impl.parser.ColumnTypeParser(
^
symbol: class ColumnTypeParser
location: package org.apache.beam.sdk.io.clickhouse.impl.parser
…/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/TableSchema.java:248:
error: cannot find symbol
new
org.apache.beam.sdk.io.clickhouse.impl.parser.ColumnTypeParser(
^
symbol: class ColumnTypeParser
location: package org.apache.beam.sdk.io.clickhouse.impl.parser
2 errors
> Task :beam-sdks-java-io-clickhouse:compileJava FAILED
```
On Sun, Feb 17, 2019 at 4:00 PM Ryan Williams <r...@runsascoded.com> wrote:
> I just captured a debug log while hitting this issue, running: `./gradlew
> -d :beam-sdks-java-extensions-sql:compileJava`
>
> The log is here
> <https://gist.githubusercontent.com/ryan-williams/1c0178b7a0f33a25c62a5d5c07d690fa/raw/05f74ca200149bd04e7aa6426bf7de9a651b0a6d/gistfile1.txt>.
> It's 18MB and doesn't really load in Chrome for me; I'd recommend wget'ing
> it.
>
> I haven't had a chance to look at it too deeply yet.
>
> On Sun, Feb 17, 2019 at 1:49 PM Ryan Williams <r...@runsascoded.com>
> wrote:
>
>> Good to know, Reuven!
>>
>> I just hit it again with `--console=verbose`, and have more info:
>> https://gist.github.com/ryan-williams/252597d9ecd6973378d5cc2e9dabceb1
>>
>> It seems that the generated files are correctly in place when the failing
>> compileJava is invoked.
>>
>> Perhaps something is wrong (and nondeterministic) with the classpath of
>> the `:beam-sdks-java-extensions-sql:compileJava` task?
>>
>> I'll see if I can run with more verbose/DEBUG logging and get any more
>> info.
>>
>> On Sun, Feb 17, 2019 at 1:42 PM Reuven Lax <re...@google.com> wrote:
>>
>>> I also see this almost every other run.
>>>
>>> On Sun, Feb 17, 2019 at 8:24 AM Ryan Williams <r...@runsascoded.com>
>>> wrote:
>>>
>>>> I still see this about 20% of the time I run `./gradlew compileTestJava
>>>> spotlessApply checkstyleMain checkstyleTest` (with 8-way parallelism),
>>>> which I've taken to running as a kind of "pre-PreCommit" (for Java).
>>>>
>>>> In total I'd estimate I've seen this error 20 times (across multiple
>>>> `./gradlew clean`s and spanning several weeks), but never twice in a row: I
>>>> always immediately re-run, and the error goes away.
>>>>
>>>> That seems consistent with the idea that gradle is racing the codegen
>>>> task against the compile task that requires the codegen's outputs.
>>>>
>>>> However, `./gradlew -p sdks/java/extensions/sql -m compileJava | grep
>>>> compileJavacc` implies that Gradle knows that compilation depends on the
>>>> codegen having run.
>>>>
>>>> Here are three successive invocations of `./gradlew compileTestJava
>>>> spotlessApply checkstyleMain checkstyleTest`
>>>> <https://gist.github.com/ryan-williams/6d2735b2484e9459be656abf379533c5>
>>>> ;
>>>>
>>>> - The first one succeeds (and includes output showing that the
>>>> codegen task, `:beam-sdks-java-extensions-sql:compileJavacc`, was run).
>>>> - Then, I rebased on one new upstream commit and ran again, and saw
>>>> the failure.
>>>> - Immediately re-running saw the failure go away.
>>>>
>>>> I guess in that output, tasks that are run but produce no stdout/stderr
>>>> are elided from the output, so we can't completely tell when the codegen
>>>> and compile tasks in question are being run. I'll start running with
>>>> --console=verbose
>>>> <https://stackoverflow.com/questions/45883963/gradle-4-0-does-not-display-executed-tasks-in-command-line#comment92914258_45889966>,
>>>> which I think will give a better sense of how this is happening.
>>>>
>>>>
>>>> On Fri, Feb 8, 2019 at 12:41 AM Ryan Williams <r...@runsascoded.com>
>>>> wrote:
>>>>
>>>>> After your last message, Alex, I saw the issue on a branch with
>>>>> minimal unrelated changes on top of b4b5495307
>>>>> <https://github.com/apache/beam/commit/b4b549530730d9e0283ad0cdb203d384a107dfbf>
>>>>> (January
>>>>> 28).
>>>>>
>>>>> I ran `./gradlew clean` for the first time in a while at that time,
>>>>> and worked happily for 11 days, but just saw the issue again on a branch
>>>>> rebased on top of 381ab55b59
>>>>> <https://github.com/apache/beam/commit/381ab55b59> (today):
>>>>>
>>>>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED
>>>>> …/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29:
>>>>> error: cannot find symbol
>>>>> import
>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
>>>>> ^
>>>>> symbol: class BeamSqlParserImpl
>>>>> location: package org.apache.beam.sdk.extensions.sql.impl.parser.impl
>>>>> 1 error
>>>>>
>>>>> Every time I've seen it, I immediately re-run the compile, and it
>>>>> succeeds.
>>>>>
>>>>> Perhaps something is still wrong with my environment, but otherwise it
>>>>> would seem that something is still flaky here.
>>>>>
>>>>> I'm compiling on an 8-core macOS machine, fwiw, and usually running
>>>>> `./gradlew compileTestJava` which compiles many projects concurrently.
>>>>>
>>>>> On Mon, Jan 28, 2019 at 4:12 PM Alex Amato <ajam...@google.com> wrote:
>>>>>
>>>>>> If it continues to occur, maybe it is an environmental issue, be sure
>>>>>> to try to clean as well.
>>>>>> ./gradlew clean
>>>>>>
>>>>>> On Mon, Jan 28, 2019 at 11:12 AM Ryan Williams <r...@runsascoded.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yea I was rebased on top of a more recent master than your previous
>>>>>>> message, when I saw it again. Perhaps I was mistaken. I'll ping here if
>>>>>>> I
>>>>>>> see it again, thanks.
>>>>>>>
>>>>>>> On Mon, Jan 28, 2019 at 1:52 PM Alex Amato <ajam...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> After I did a rebase, it went away for me. So I think that this
>>>>>>>> should work. Are you saying that you did rebase ontop of master and it
>>>>>>>> still occurred? Strange.
>>>>>>>>
>>>>>>>> On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams <r...@runsascoded.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hm, I just encountered this again on a branch that based on
>>>>>>>>> 5b46b02b49 (top of trunk from this afternoon). Is it definitely
>>>>>>>>> supposed to
>>>>>>>>> be fixed?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato <ajam...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Please try rebasing from master, I believe this issue has been
>>>>>>>>>> resolved.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams <
>>>>>>>>>> r...@runsascoded.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm seeing every ≈third `./gradlew compileJava` fail locally due
>>>>>>>>>>> to this; re-running the commit has always succeeded, so far.
>>>>>>>>>>>
>>>>>>>>>>> Sounds like there is not an immediate fix in the works / no one
>>>>>>>>>>> assigned on the JIRA?
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles <k...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> This might connect to vendoring Calcite. It will be easiest,
>>>>>>>>>>>> and have the best incremental build, if we separate the generated
>>>>>>>>>>>> code into
>>>>>>>>>>>> its own module that has relocation to match the vendored Calcite.
>>>>>>>>>>>>
>>>>>>>>>>>> Kenn
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin <ke...@google.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> We don't pre-generate the code as a separate step. Code gen
>>>>>>>>>>>>> from the SQL parser syntax spec and its compilation happens both
>>>>>>>>>>>>> during the
>>>>>>>>>>>>> Beam SQL build task. Splitting the code generation and
>>>>>>>>>>>>> compilation might
>>>>>>>>>>>>> not be trivial. We definitely should look into fixing this though.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Anton
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato <
>>>>>>>>>>>>> ajam...@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Okay, make sense perhaps we can somehow make it fail when it
>>>>>>>>>>>>>> fails to generate the dep, rather than when compiling the java
>>>>>>>>>>>>>> code later on
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin <
>>>>>>>>>>>>>> ke...@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ParserImpl is autogenerated by Calcite at build time. It
>>>>>>>>>>>>>>> seems that there's a race condition there and it sometimes
>>>>>>>>>>>>>>> fails. Rerunning
>>>>>>>>>>>>>>> the build works for me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Anton
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jan 23, 2019, 11:06 AM Alex Amato <
>>>>>>>>>>>>>>> ajam...@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://jira.apache.org/jira/browse/BEAM-6495?filter=-2
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any ideas, how this got through the precommit?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /usr/local/google/home/ajamato/go/src/
>>>>>>>>>>>>>>>> github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29:
>>>>>>>>>>>>>>>> error: cannot find symbol
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ^
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> symbol: class BeamSqlParserImpl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> location: package
>>>>>>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1 error
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
