[jira] [Commented] (HIVE-17822) Provide an option to skip shading of jars
[ https://issues.apache.org/jira/browse/HIVE-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208894#comment-16208894 ] Lefty Leverenz commented on HIVE-17822: --- Should this be documented in the wiki? (If so, it needs a TODOC3.0 label.) > Provide an option to skip shading of jars > - > > Key: HIVE-17822 > URL: https://issues.apache.org/jira/browse/HIVE-17822 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-17822.1.patch > > > Maven shade plugin does not have option to skip. Adding it under a profile > can help with skip shade reducing build times. > Maven build profile shows druid and jdbc shade plugin to be slowest (also > hive-exec). For devs not working on druid or jdbc, it will be good to have an > option to skip shading via a profile. With this it will be possible to get a > subminute dev build. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17822) Provide an option to skip shading of jars
[ https://issues.apache.org/jira/browse/HIVE-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207813#comment-16207813 ] Ashutosh Chauhan commented on HIVE-17822: - +1 > Provide an option to skip shading of jars > - > > Key: HIVE-17822 > URL: https://issues.apache.org/jira/browse/HIVE-17822 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17822.1.patch > > > Maven shade plugin does not have option to skip. Adding it under a profile > can help with skip shade reducing build times. > Maven build profile shows druid and jdbc shade plugin to be slowest (also > hive-exec). For devs not working on druid or jdbc, it will be good to have an > option to skip shading via a profile. With this it will be possible to get a > subminute dev build. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17822) Provide an option to skip shading of jars
[
https://issues.apache.org/jira/browse/HIVE-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206948#comment-16206948
]
Hive QA commented on HIVE-17822:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892483/HIVE-17822.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11242 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=243)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94]
(batchId=241)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7337/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7337/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7337/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892483 - PreCommit-HIVE-Build
> Provide an option to skip shading of jars
> -
>
> Key: HIVE-17822
> URL: https://issues.apache.org/jira/browse/HIVE-17822
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17822.1.patch
>
>
> Maven shade plugin does not have option to skip. Adding it under a profile
> can help with skip shade reducing build times.
> Maven build profile shows druid and jdbc shade plugin to be slowest (also
> hive-exec). For devs not working on druid or jdbc, it will be good to have an
> option to skip shading via a profile. With this it will be possible to get a
> subminute dev build.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17822) Provide an option to skip shading of jars
[ https://issues.apache.org/jira/browse/HIVE-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206768#comment-16206768 ] Prasanth Jayachandran commented on HIVE-17822: -- [~ashutoshc] Can you please review this change? > Provide an option to skip shading of jars > - > > Key: HIVE-17822 > URL: https://issues.apache.org/jira/browse/HIVE-17822 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17822.1.patch > > > Maven shade plugin does not have option to skip. Adding it under a profile > can help with skip shade reducing build times. > Maven build profile shows druid and jdbc shade plugin to be slowest (also > hive-exec). For devs not working on druid or jdbc, it will be good to have an > option to skip shading via a profile. With this it will be possible to get a > subsecond dev build. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17822) Provide an option to skip shading of jars
[
https://issues.apache.org/jira/browse/HIVE-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206727#comment-16206727
]
Prasanth Jayachandran commented on HIVE-17822:
--
Some numbers
{code}
# Base: Clean offline quite build
$ time mvn clean install -DskipTests -o -q
real3m9.005s
user7m14.864s
sys 0m40.295s
# Parallel (using 1C gave best build times) build
$ time mvn clean install -DskipTests -T 1C -o -q
real2m24.415s
user8m12.243s
sys 0m54.905s
# With MAVEN_OPTS
$ time MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1" mvn clean
install -DskipTests -T 1C -o -q
real2m12.872s
user7m46.879s
sys 0m49.696s
# Skip clean
$ MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
$ time MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1" mvn install
-DskipTests -T 1C -o -q
real1m31.403s
user5m13.439s
sys 0m37.885s
# Skip shade for jdbc and druid-handler (requires HIVE-17822)
# NOTE: if you are changing/testing jdbc or druid you may want to skip this step
$ time MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1" mvn install
-DskipShade -DskipTests -T 1C -o -q
real1m20.130s
user4m37.645s
sys 0m39.897s
# Skip remote resource plugin
$ time MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1" mvn install
-DskipShade -DskipTests -Dremoteresources.skip=true -T 1C -o -q
real0m37.485s
user0m52.652s
sys 0m14.118s
# Build ql and downstream modules
$ time MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1" mvn install
-DskipShade -DskipTests -Dremoteresources.skip=true -T 1C -o -q -pl ql -amd
real0m31.827s
user1m50.349s
sys 0m9.494s
# Build llap-server and downstream modules
$ time MAVEN_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1" mvn install
-DskipShade -DskipTests -Dremoteresources.skip=true -T 1C -o -q -pl llap-server
-amd
real0m9.147s
user0m20.189s
sys 0m3.056s
{code}
> Provide an option to skip shading of jars
> -
>
> Key: HIVE-17822
> URL: https://issues.apache.org/jira/browse/HIVE-17822
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17822.1.patch
>
>
> Maven shade plugin does not have option to skip. Adding it under a profile
> can help with skip shade reducing build times.
> Maven build profile shows druid and jdbc shade plugin to be slowest (also
> hive-exec). For devs not working on druid or jdbc, it will be good to have an
> option to skip shading via a profile. With this it will be possible to get a
> subsecond dev build.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17822) Provide an option to skip shading of jars
[ https://issues.apache.org/jira/browse/HIVE-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206701#comment-16206701 ] Prasanth Jayachandran commented on HIVE-17822: -- Users have to pass -DskipShade to skip druid and jdbc shading. hive-exec shading is not used as submodules depend on it in most cases. > Provide an option to skip shading of jars > - > > Key: HIVE-17822 > URL: https://issues.apache.org/jira/browse/HIVE-17822 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17822.1.patch > > > Maven shade plugin does not have option to skip. Adding it under a profile > can help with skip shade reducing build times. > Maven build profile shows druid and jdbc shade plugin to be slowest (also > hive-exec). For devs not working on druid or jdbc, it will be good to have an > option to skip shading via a profile. With this it will be possible to get a > subsecond dev build. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
