[
https://issues.apache.org/jira/browse/IMPALA-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814469#comment-17814469
]
Joe McDonnell commented on IMPALA-9153:
---------------------------------------
I did a little digging and here is what I found. Impala uses the CZT version of
CUP (and the cup-maven-plugin). It turns out that we need that. The regular
version of CUP generates Java code in a way that won't compile if the grammar
is too big (it gets "code too large" errors from Java compilation). The CZT
version of CUP avoids that by generating code with a separate additional file
containing the action table. The generated code reads that extra file rather
than having the array hard-coded in the Java code itself. It has a couple other
similar enhancements. Here is a commit that is in CZT that isn't in vanilla
CUP:
[https://sourceforge.net/p/czt/code/ci/e359ed1f896f5487c215d749f25cc3a07e956876/]
Long story short: regular CUP won't work for us.
The CZT code is located here:
[https://sourceforge.net/p/czt/code/ci/master/tree/dev/] It has three parts:
java-cup, java-cup-runtime, and cup-maven-plugin. If we were to patch
cup-maven-plugin to avoid regeneration when the generated files are up to date,
it would solve this. I tried it locally and it works. Here is the code for the
plugin:
[https://sourceforge.net/p/czt/code/ci/master/tree/dev/cup-maven-plugin/src/main/java/net/sourceforge/czt/cup/maven/CupGenerateMojo.java]
So, we have a couple options:
# We vendor the cup-maven-plugin and modify it to refer to the existing CUP
artifacts that we use (java-cup, java-cup-runtime).
# We vendor the cup-maven-plugin as well as java-cup and java-cup-runtime. CZT
is not active, and it might be easier for us to have the code right there. The
artifact we currently use for CUP is some random CDH build of CUP.
All the licenses are ASF compatible.
> Avoid unneccessary cup compilation
> ----------------------------------
>
> Key: IMPALA-9153
> URL: https://issues.apache.org/jira/browse/IMPALA-9153
> Project: IMPALA
> Issue Type: Test
> Components: Infrastructure
> Reporter: Quanlong Huang
> Priority: Major
>
> Everytime when I run a single FE test, maven always recompile the whole
> module since it detects changes. The change is made by cup-maven-plugin. It
> recompiles sql-parser.cup to SqlParser.java and SqlParserSymbols.java
> everytime even there're no changes in sql-parser.cup. The recompilation cost
> some time and slow down the reproducing of some flasky test that requires
> running it many times.
> An output example:
> {code:java}
> $ (pushd fe && mvn test -fae -Dtest=AuthorizationStmtTest#test*Enabled[1])
> ~/workspace/Impala/fe ~/workspace/Impala
> [INFO] Scanning for projects...
> [INFO]
> [INFO] -----------------< org.apache.impala:impala-frontend
> >------------------
> [INFO] Building Apache Impala Query Engine Frontend 0.1-SNAPSHOT
> [INFO] --------------------------------[ jar
> ]---------------------------------
> [WARNING] Could not transfer metadata
> com.cloudera.cdh:cdh-root:6.x-SNAPSHOT/maven-metadata.xml from/to
> ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}): Cannot access
> ${distMgmtSnapshotsUrl} with type default using the available connector
> factories: BasicRepositoryConnectorFactory
> [INFO]
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce
> (enforce-banned-dependencies) @ impala-frontend ---
> [INFO]
> [INFO] --- jacoco-maven-plugin:0.7.6.201602180812:prepare-agent
> (prepare-jacoco-agent) @ impala-frontend ---
> [INFO] Skipping JaCoCo execution because property jacoco.skip is set.
> [INFO] surefireJacocoArg set to empty
> [INFO]
> [INFO] --- cup-maven-plugin:1.6-cdh:generate (cup) @ impala-frontend ---
> [INFO] CUP: Processing 1 cup files <------------ Always regenerate the files
> Warning : Terminal "EMPTY_IDENT" was declared but never used
> Warning : Terminal "UNEXPECTED_CHAR" was declared but never used
> ------- CUP v0.11a czt01 beta Parser Generation Summary -------
> 0 errors and 2 warnings
> 229 terminals, 228 non-terminals, and 875 productions declared,
> producing 1620 unique parse states.
> 2 terminals declared but not used.
> 0 non-terminals declared but not used.
> 0 productions never reduced.
> 0 conflicts detected (0 expected).
> Code written to "SqlParser.java", and "SqlParserSymbols.java".
> ---------------------------------------------------- (v0.11a czt01 beta)
> [WARNING] /home/quanlong/workspace/Impala/fe/src/main/cup/sql-parser.cup
> [0:0]: Terminal "EMPTY_IDENT" was declared but never used
> [WARNING] /home/quanlong/workspace/Impala/fe/src/main/cup/sql-parser.cup
> [0:0]: Terminal "UNEXPECTED_CHAR" was declared but never used
> [INFO] CUP: generated
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup/org/apache/impala/analysis/SqlParser.java
> [INFO] CUP: generated
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup/org/apache/impala/analysis/SqlParserSymbols.java
> [INFO] CUP: generated
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup/org/apache/impala/analysis/action_table.dat
> [INFO]
> [INFO] --- maven-jflex-plugin:1.4.3:generate (jflex) @ impala-frontend ---
> [INFO] SqlScanner.java is up to date.
> [INFO]
> [INFO] --- maven-dependency-plugin:3.1.1:build-classpath (write-classpath) @
> impala-frontend ---
> [INFO] Skipped writing classpath file
> '/home/quanlong/workspace/Impala/fe/target/build-classpath.txt'. No changes
> found.
> [INFO]
> [INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @
> impala-frontend ---
> [INFO] Source directory:
> /home/quanlong/workspace/Impala/fe/generated-sources/gen-java added.
> [INFO] Source directory:
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup added.
> [INFO] Source directory:
> /home/quanlong/workspace/Impala/fe/src/compat-hive-2/java added.
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> impala-frontend ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory
> /home/quanlong/workspace/Impala/fe/src/main/resources
> [INFO] Copying 1 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @
> impala-frontend ---
> [INFO] Changes detected - recompiling the module! <--------- Decide to
> recomple
> [INFO] Compiling 907 source files to
> /home/quanlong/workspace/Impala/fe/target/classes
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[24,16]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[24,16]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[24,16]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[33,23]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[40,15]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[45,25]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java:
> Some input files use or override a deprecated API.
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java:
> Recompile with -Xlint:deprecation for details.
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/analysis/Expr.java:
> Some input files use unchecked or unsafe operations.
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/analysis/Expr.java:
> Recompile with -Xlint:unchecked for details.
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
> impala-frontend ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 43 resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @
> impala-frontend ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- maven-surefire-plugin:2.20:test (default-test) @ impala-frontend
> ---
> [INFO]
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.impala.authorization.AuthorizationStmtTest
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.847
> s - in org.apache.impala.authorization.AuthorizationStmtTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
> [INFO] --- jacoco-maven-plugin:0.7.6.201602180812:report (jacoco-report) @
> impala-frontend ---
> [INFO] Skipping JaCoCo execution because property jacoco.skip is set.
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 01:23 min
> [INFO] Finished at: 2019-11-13T15:50:33+08:00
> [INFO]
> ------------------------------------------------------------------------
> {code}
> We should configure cup-maven-plugin to avoid unneccessary recompilation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
