[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842371#comment-15842371 ] Hive QA commented on HIVE-15669: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849060/HIVE-15669.3.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10988 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=114) [bucketmapjoin3.q,load_dyn_part5.q,union_date.q,cbo_gby.q,auto_join31.q,auto_sortmerge_join_1.q,join_cond_pushdown_unqual1.q,ppd_outer_join3.q,bucket_map_join_spark3.q,union28.q,statsfs.q,escape_sortby1.q,leftsemijoin.q,union_remove_6.q,join29.q] org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3209/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3209/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3209/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12849060 - PreCommit-HIVE-Build > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch, HIVE-15669.1.patch, > HIVE-15669.2.patch, HIVE-15669.3.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840584#comment-15840584 ] Rajesh Balamohan commented on HIVE-15669: - Thanks for the patch [~prasanth_j]. LGTM. +1. > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch, HIVE-15669.1.patch, > HIVE-15669.2.patch, HIVE-15669.3.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833797#comment-15833797 ] Prasanth Jayachandran commented on HIVE-15669: -- I am not sure what would be correct delta here. Should it be wrt to dag start time or attempt start time? Don't starve the dag for long time as opposed to don't starve the attempt. My thinking was not starving the dag for a long time. > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833777#comment-15833777 ] Rajesh Balamohan commented on HIVE-15669: - [~prasanth_j] - Should the comparison be based on "getFirstAttemptStartTime" as opposed to "getDagStartTime" ? > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15669.1.patch > > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832822#comment-15832822 ] Hive QA commented on HIVE-15669: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12848458/HIVE-15669.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3086/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3086/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3086/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-01-21 05:33:23.425 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3086/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-01-21 05:33:23.427 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at d9343f6 HIVE-15544 : Support scalar subqueries (Vineet Garg via Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at d9343f6 HIVE-15544 : Support scalar subqueries (Vineet Garg via Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-01-21 05:33:24.446 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: patch -p1 patching file llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java patching file llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java patching file llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestShortestJobFirstComparator.java + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java does not exist: must build /data/hiveptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g org/apache/hadoop/hive/metastore/parser/Filter.g DataNucleus Enhancer (version 4.1.6) for API "JDO" DataNucleus Enhancer : Classpath >> /usr/share/maven/boot/plexus-classworlds-2.x.jar ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MColumnDescriptor ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStorageDescriptor ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartition ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MIndex ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRole ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRoleMap ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MGlobalPrivilege ENHANCED (Persistable) :
[jira] [Commented] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler
[ https://issues.apache.org/jira/browse/HIVE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830937#comment-15830937 ] Prasanth Jayachandran commented on HIVE-15669: -- cc [~rajesh.balamohan] > LLAP: Improve aging in shortest job first scheduler > --- > > Key: HIVE-15669 > URL: https://issues.apache.org/jira/browse/HIVE-15669 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > > Under high concurrency, some jobs can gets starved for longer time when > hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for > locality). -- This message was sent by Atlassian JIRA (v6.3.4#6332)