[jira] [Commented] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946501#comment-16946501
]
Yuming Wang commented on HIVE-22097:
Could we backport this patch to branch-3.0 and branch-3.1?
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946492#comment-16946492
]
Hive QA commented on HIVE-22255:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 21 unchanged - 0 fixed
= 23 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18901/dev-support/hive-personality.sh
|
| git revision | master / 050f918 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18901/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18901/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18901/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22255.patch
>
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
[jira] [Commented] (HIVE-22235) CommandProcessorResponse should not be an exception
[
https://issues.apache.org/jira/browse/HIVE-22235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946479#comment-16946479
]
Hive QA commented on HIVE-22235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982419/HIVE-22235.08.patch
{color:green}SUCCESS:{color} +1 due to 75 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17305 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=234)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=234)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18900/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18900/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18900/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982419 - PreCommit-HIVE-Build
> CommandProcessorResponse should not be an exception
> ---
>
> Key: HIVE-22235
> URL: https://issues.apache.org/jira/browse/HIVE-22235
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22235.01.patch, HIVE-22235.02.patch,
> HIVE-22235.03.patch, HIVE-22235.04.patch, HIVE-22235.05.patch,
> HIVE-22235.06.patch, HIVE-22235.07.patch, HIVE-22235.08.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The CommandProcessorResponse class extends Exception. This may be convenient,
> but it is wrong, as a response is not an exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22235) CommandProcessorResponse should not be an exception
[
https://issues.apache.org/jira/browse/HIVE-22235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946474#comment-16946474
]
Hive QA commented on HIVE-22235:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} cli in master has 9 extant Findbugs warnings. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
49s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 73 new + 1432 unchanged - 467
fixed = 1505 total (was 1899) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} cli: The patch generated 0 new + 39 unchanged - 2
fixed = 39 total (was 41) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/core: The patch generated 0 new + 75
unchanged - 16 fixed = 75 total (was 91) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 183 unchanged - 11 fixed = 183 total (was 194) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch java-client
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[
https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946434#comment-16946434
]
Hive QA commented on HIVE-22136:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982418/HIVE-22136.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17515 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=299)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18899/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18899/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18899/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982418 - PreCommit-HIVE-Build
> Turn on tez.bucket.pruning
> ---
>
> Key: HIVE-22136
> URL: https://issues.apache.org/jira/browse/HIVE-22136
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch,
> HIVE-22136.3.patch, HIVE-22136.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22274) Upgrade Calcite version to 1.21.0
[ https://issues.apache.org/jira/browse/HIVE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Carlin updated HIVE-22274: Attachment: HIVE-22274.2.patch > Upgrade Calcite version to 1.21.0 > - > > Key: HIVE-22274 > URL: https://issues.apache.org/jira/browse/HIVE-22274 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch, HIVE-22274.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22301) Hive lineage is not generated for insert overwrite queries on partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-22301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated HIVE-22301: -- Description: Problem: When I run the below mentioned queries, the last query should have given the proper hive lineage info (through HookContext) from table_b to table_t. * Create table table_t (id int) partitioned by (dob date); * Create table table_b (id int) partitioned by (dob date); * from table_b a insert overwrite table table_t select a.id,a.dob; Note : for CTAS query from a partitioned table , this issue is not seen. Only for insert queries like insert into select * from and query like above, issue is seen. Technical Observations: At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot from IntelliJ. !ScreenShot RunPostExecHook.png|width=728,height=427! I found that the PrivateHookContext is getting created with proper outputs value as shown below initially: !ScreenShot HookContext.png|width=714,height=541! The same is passed properly to runBeforeExecutionHook as shown below: !ScreenShot runBeforeExecution.png|width=719,height=620! Later when we pass HookContext to runPostExecHooks, there is no output populated. Kindly check the reason and let me know if you need any further information from my end. was: Problem: When I run the below mentioned queries, the last query should have given the proper hive lineage info (through HookContext) from table_b to table_t. * Create table table_t (id int) partitioned by (dob date); * Create table table_b (id int) partitioned by (dob date); * from table_b a insert overwrite table table_t select a.id,a.dob; Note : for CTAS query from a partitioned table , this issue is not seen. Only for insert queries like insert into select * from and query like above, issue is seen. This issue is seen in latest HDP builds as well. Technical Observations: At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot from IntelliJ. !ScreenShot RunPostExecHook.png|width=728,height=427! I found that the PrivateHookContext is getting created with proper outputs value as shown below initially: !ScreenShot HookContext.png|width=714,height=541! The same is passed properly to runBeforeExecutionHook as shown below: !ScreenShot runBeforeExecution.png|width=719,height=620! Later when we pass HookContext to runPostExecHooks, there is no output populated. Kindly check the reason and let me know if you need any further information from my end. > Hive lineage is not generated for insert overwrite queries on partitioned > tables > > > Key: HIVE-22301 > URL: https://issues.apache.org/jira/browse/HIVE-22301 > Project: Hive > Issue Type: Bug > Components: lineage >Affects Versions: 3.1.2 >Reporter: Sidharth Kumar Mishra >Priority: Major > Attachments: ScreenShot HookContext.png, ScreenShot > RunPostExecHook.png, ScreenShot runBeforeExecution.png > > > Problem: When I run the below mentioned queries, the last query should have > given the proper hive lineage info (through HookContext) from table_b to > table_t. > * Create table table_t (id int) partitioned by (dob date); > * Create table table_b (id int) partitioned by (dob date); > * from table_b a insert overwrite table table_t select a.id,a.dob; > Note : for CTAS query from a partitioned table , this issue is not seen. Only > for insert queries like insert into select * from and query > like above, issue is seen. > > Technical Observations: > At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through > hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot > from IntelliJ. > !ScreenShot RunPostExecHook.png|width=728,height=427! > > I found that the PrivateHookContext is getting created with proper outputs > value as shown below initially: > !ScreenShot HookContext.png|width=714,height=541! > The same is passed properly to runBeforeExecutionHook as shown below: > !ScreenShot runBeforeExecution.png|width=719,height=620! > > Later when we pass HookContext to runPostExecHooks, there is no output > populated. Kindly check the reason and let me know if you need any further > information from my end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[
https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946400#comment-16946400
]
Hive QA commented on HIVE-22136:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18899/dev-support/hive-personality.sh
|
| git revision | master / 050f918 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18899/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18899/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Turn on tez.bucket.pruning
> ---
>
> Key: HIVE-22136
> URL: https://issues.apache.org/jira/browse/HIVE-22136
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch,
> HIVE-22136.3.patch, HIVE-22136.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22301) Hive lineage is not generated for insert overwrite queries on partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-22301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated HIVE-22301: -- Component/s: lineage > Hive lineage is not generated for insert overwrite queries on partitioned > tables > > > Key: HIVE-22301 > URL: https://issues.apache.org/jira/browse/HIVE-22301 > Project: Hive > Issue Type: Bug > Components: lineage >Affects Versions: 3.1.2 >Reporter: Sidharth Kumar Mishra >Priority: Major > Attachments: ScreenShot HookContext.png, ScreenShot > RunPostExecHook.png, ScreenShot runBeforeExecution.png > > > Problem: When I run the below mentioned queries, the last query should have > given the proper hive lineage info (through HookContext) from table_b to > table_t. > * Create table table_t (id int) partitioned by (dob date); > * Create table table_b (id int) partitioned by (dob date); > * from table_b a insert overwrite table table_t select a.id,a.dob; > Note : for CTAS query from a partitioned table , this issue is not seen. Only > for insert queries like insert into select * from and query > like above, issue is seen. > This issue is seen in latest HDP builds as well. > > Technical Observations: > At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through > hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot > from IntelliJ. > !ScreenShot RunPostExecHook.png|width=728,height=427! > > I found that the PrivateHookContext is getting created with proper outputs > value as shown below initially: > !ScreenShot HookContext.png|width=714,height=541! > The same is passed properly to runBeforeExecutionHook as shown below: > !ScreenShot runBeforeExecution.png|width=719,height=620! > > Later when we pass HookContext to runPostExecHooks, there is no output > populated. Kindly check the reason and let me know if you need any further > information from my end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946392#comment-16946392 ] LuGuangMing commented on HIVE-22198: [~aditya-shah] ConditionalTask is not easily aware of the parent node that needs to be repaired, such as the following SQL scenario: create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; !image-2019-10-08-09-51-06-144.png! Its scheduling order selects a branch within ConditionalTask, The parent relationship of a branch that is selected for execution can be preserved, and the parent relationship of a branch that is not selected for execution must be deleted, The branch that we want to execute is not easily perceived in advance, So I consider placing dependency repair on the parent node in the launchTask phase > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, image-2019-10-08-09-51-06-144.png, > test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22301) Hive lineage is not generated for insert overwrite queries on partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-22301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated HIVE-22301: -- Affects Version/s: 3.1.2 > Hive lineage is not generated for insert overwrite queries on partitioned > tables > > > Key: HIVE-22301 > URL: https://issues.apache.org/jira/browse/HIVE-22301 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Sidharth Kumar Mishra >Priority: Major > Attachments: ScreenShot HookContext.png, ScreenShot > RunPostExecHook.png, ScreenShot runBeforeExecution.png > > > Problem: When I run the below mentioned queries, the last query should have > given the proper hive lineage info (through HookContext) from table_b to > table_t. > * Create table table_t (id int) partitioned by (dob date); > * Create table table_b (id int) partitioned by (dob date); > * from table_b a insert overwrite table table_t select a.id,a.dob; > Note : for CTAS query from a partitioned table , this issue is not seen. Only > for insert queries like insert into select * from and query > like above, issue is seen. > This issue is seen in latest HDP builds as well. > > Technical Observations: > At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through > hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot > from IntelliJ. > !ScreenShot RunPostExecHook.png|width=728,height=427! > > I found that the PrivateHookContext is getting created with proper outputs > value as shown below initially: > !ScreenShot HookContext.png|width=714,height=541! > The same is passed properly to runBeforeExecutionHook as shown below: > !ScreenShot runBeforeExecution.png|width=719,height=620! > > Later when we pass HookContext to runPostExecHooks, there is no output > populated. Kindly check the reason and let me know if you need any further > information from my end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Attachment: image-2019-10-08-09-51-06-144.png > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, image-2019-10-08-09-51-06-144.png, > test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?focusedWorklogId=324802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324802 ] ASF GitHub Bot logged work on HIVE-22252: - Author: ASF GitHub Bot Created on: 08/Oct/19 01:50 Start Date: 08/Oct/19 01:50 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #799: HIVE-22252: Fix caught NullPointerExceptions generated during EXPLAIN URL: https://github.com/apache/hive/pull/799 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324802) Time Spent: 20m (was: 10m) > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch, > HIVE-22252.3.patch, HIVE-22252.4.patch > > Time Spent: 20m > Remaining Estimate: 0h > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22252: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~jfs]! > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch, > HIVE-22252.3.patch, HIVE-22252.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946388#comment-16946388
]
Hive QA commented on HIVE-22252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982405/HIVE-22252.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17515 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18898/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18898/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18898/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982405 - PreCommit-HIVE-Build
> Fix caught NullPointerExceptions generated during EXPLAIN
> -
>
> Key: HIVE-22252
> URL: https://issues.apache.org/jira/browse/HIVE-22252
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch,
> HIVE-22252.3.patch, HIVE-22252.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While debugging an issue I noticed that during EXPLAIN the following methods
> throw a NullPointerException:
> VectorColumnOutputMapping#finalize
> AbstractOperatorDesc#getUserLevelStatistics
> AbstractOperatorDesc#getColumnExprMapForExplain
> The exceptions do end up getting caught but we should add null checks and
> gracefully to be less wasteful and to aid future debugging.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22239) Scale data size using column value ranges
[ https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22239: --- Attachment: HIVE-22239.02.patch > Scale data size using column value ranges > - > > Key: HIVE-22239 > URL: https://issues.apache.org/jira/browse/HIVE-22239 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, > HIVE-22239.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, min/max values for columns are only used to determine whether a > certain range filter falls out of range and thus filters all rows or none at > all. If it does not, we just use a heuristic that the condition will filter > 1/3 of the input rows. Instead of using that heuristic, we can use another > one that assumes that data will be uniformly distributed across that range, > and calculate the selectivity for the condition accordingly. > This patch also includes the propagation of min/max column values from > statistics to the optimizer for timestamp type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22239) Scale data size using column value ranges
[ https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946385#comment-16946385 ] Jesus Camacho Rodriguez commented on HIVE-22239: [~kgyrtkirk], [~mgergely], could you take a look? https://github.com/apache/hive/pull/787 Thanks > Scale data size using column value ranges > - > > Key: HIVE-22239 > URL: https://issues.apache.org/jira/browse/HIVE-22239 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, > HIVE-22239.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, min/max values for columns are only used to determine whether a > certain range filter falls out of range and thus filters all rows or none at > all. If it does not, we just use a heuristic that the condition will filter > 1/3 of the input rows. Instead of using that heuristic, we can use another > one that assumes that data will be uniformly distributed across that range, > and calculate the selectivity for the condition accordingly. > This patch also includes the propagation of min/max column values from > statistics to the optimizer for timestamp type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-22255:
--
Attachment: HIVE-22255.patch
Status: Patch Available (was: Open)
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22255.patch
>
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 |
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 |
> ++{code}
>
> you will see that Major compaction will not be trigger until you run alter
> table compact MAJOR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946359#comment-16946359
]
Hive QA commented on HIVE-22252:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 1
fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18898/dev-support/hive-personality.sh
|
| git revision | master / e16877c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18898/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18898/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix caught NullPointerExceptions generated during EXPLAIN
> -
>
> Key: HIVE-22252
> URL: https://issues.apache.org/jira/browse/HIVE-22252
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch,
> HIVE-22252.3.patch, HIVE-22252.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While debugging an issue I noticed that during EXPLAIN the following methods
> throw a NullPointerException:
> VectorColumnOutputMapping#finalize
> AbstractOperatorDesc#getUserLevelStatistics
> AbstractOperatorDesc#getColumnExprMapForExplain
> The exceptions do end up getting caught but we should add null checks and
> gracefully to be less wasteful and to aid future debugging.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22239) Scale data size using column value ranges
[
https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946350#comment-16946350
]
Hive QA commented on HIVE-22239:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982406/HIVE-22239.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 17516 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_type]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge5] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge6] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge5]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge6]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge7]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=111)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=137)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18897/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18897/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18897/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982406 - PreCommit-HIVE-Build
> Scale data size using column value ranges
> -
>
> Key: HIVE-22239
> URL: https://issues.apache.org/jira/browse/HIVE-22239
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22239.01.patch, HIVE-22239.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, min/max values for columns are only used to determine whether a
> certain range filter falls out of range and thus filters all rows or none at
> all. If it does not, we just use a heuristic that the condition will filter
> 1/3 of the input rows. Instead of using that heuristic, we can use another
> one that assumes that data will be uniformly distributed across that range,
> and calculate the selectivity for the condition accordingly.
> This patch also includes the propagation of min/max column values from
> statistics to the optimizer for timestamp type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22301) Hive lineage is not generated for insert overwrite queries on partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-22301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated HIVE-22301: - Description: Problem: When I run the below mentioned queries, the last query should have given the proper hive lineage info (through HookContext) from table_b to table_t. * Create table table_t (id int) partitioned by (dob date); * Create table table_b (id int) partitioned by (dob date); * from table_b a insert overwrite table table_t select a.id,a.dob; Note : for CTAS query from a partitioned table , this issue is not seen. Only for insert queries like insert into select * from and query like above, issue is seen. This issue is seen in latest HDP builds as well. Technical Observations: At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot from IntelliJ. !ScreenShot RunPostExecHook.png|width=728,height=427! I found that the PrivateHookContext is getting created with proper outputs value as shown below initially: !ScreenShot HookContext.png|width=714,height=541! The same is passed properly to runBeforeExecutionHook as shown below: !ScreenShot runBeforeExecution.png|width=719,height=620! Later when we pass HookContext to runPostExecHooks, there is no output populated. Kindly check the reason and let me know if you need any further information from my end. was: Problem: When I run the below mentioned queries, the last query should have given the proper hive lineage info (through HookContext) from table_b to table_t. * Create table table_t (id int) partitioned by (dob date); * Create table table_b (id int) partitioned by (dob date); * from table_b a insert overwrite table table_t select a.id,a.dob; Note : for CTAS query from a partitioned table , this issue is not seen. Only for insert queries like insert into select * from and query like above, issue is seen. This issue is seen in latest HDP builds as well. Technical Observations: At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot from IntelliJ. !ScreenShot RunPostExecHook.png|width=2726,height=1599! I found that the PrivateHookContext is getting created with proper outputs value as shown below initially: !ScreenShot HookContext.png! The same is passed properly to runBeforeExecutionHook as shown below: !ScreenShot runBeforeExecution.png! Later when we pass HookContext to runPostExecHooks, there is no output populated. Kindly check the reason and let me know if you need any further information from my end. > Hive lineage is not generated for insert overwrite queries on partitioned > tables > > > Key: HIVE-22301 > URL: https://issues.apache.org/jira/browse/HIVE-22301 > Project: Hive > Issue Type: Bug >Reporter: Sidharth Kumar Mishra >Priority: Major > Attachments: ScreenShot HookContext.png, ScreenShot > RunPostExecHook.png, ScreenShot runBeforeExecution.png > > > Problem: When I run the below mentioned queries, the last query should have > given the proper hive lineage info (through HookContext) from table_b to > table_t. > * Create table table_t (id int) partitioned by (dob date); > * Create table table_b (id int) partitioned by (dob date); > * from table_b a insert overwrite table table_t select a.id,a.dob; > Note : for CTAS query from a partitioned table , this issue is not seen. Only > for insert queries like insert into select * from and query > like above, issue is seen. > This issue is seen in latest HDP builds as well. > > Technical Observations: > At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through > hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot > from IntelliJ. > !ScreenShot RunPostExecHook.png|width=728,height=427! > > I found that the PrivateHookContext is getting created with proper outputs > value as shown below initially: > !ScreenShot HookContext.png|width=714,height=541! > The same is passed properly to runBeforeExecutionHook as shown below: > !ScreenShot runBeforeExecution.png|width=719,height=620! > > Later when we pass HookContext to runPostExecHooks, there is no output > populated. Kindly check the reason and let me know if you need any further > information from my end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22301) Hive lineage is not generated for insert overwrite queries on partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-22301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharth Kumar Mishra updated HIVE-22301: - Description: Problem: When I run the below mentioned queries, the last query should have given the proper hive lineage info (through HookContext) from table_b to table_t. * Create table table_t (id int) partitioned by (dob date); * Create table table_b (id int) partitioned by (dob date); * from table_b a insert overwrite table table_t select a.id,a.dob; Note : for CTAS query from a partitioned table , this issue is not seen. Only for insert queries like insert into select * from and query like above, issue is seen. This issue is seen in latest HDP builds as well. Technical Observations: At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot from IntelliJ. !ScreenShot RunPostExecHook.png|width=2726,height=1599! I found that the PrivateHookContext is getting created with proper outputs value as shown below initially: !ScreenShot HookContext.png! The same is passed properly to runBeforeExecutionHook as shown below: !ScreenShot runBeforeExecution.png! Later when we pass HookContext to runPostExecHooks, there is no output populated. Kindly check the reason and let me know if you need any further information from my end. was: Problem: When I run the below mentioned queries, the last query should have given the proper hive lineage info (through HookContext) from table_b to table_t. * Create table table_t (id int) partitioned by (dob date); * Create table table_b (id int) partitioned by (dob date); * from table_b a insert overwrite table table_t select a.id,a.dob; Note : for CTAS query from a partitioned table , this issue is not seen. Only for insert queries like insert into select * from and query like above, issue is seen. This issue is seen in latest HDP builds as well. Technical Observations: At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot from IntelliJ. !ScreenShot RunPostExecHook.png! I found that the PrivateHookContext is getting created with proper outputs value as shown below initially: !ScreenShot HookContext.png! The same is passed properly to runBeforeExecutionHook as shown below: !ScreenShot runBeforeExecution.png! Later when we pass HookContext to runPostExecHooks, there is no output populated. Kindly check the reason and let me know if you need any further information from my end. > Hive lineage is not generated for insert overwrite queries on partitioned > tables > > > Key: HIVE-22301 > URL: https://issues.apache.org/jira/browse/HIVE-22301 > Project: Hive > Issue Type: Bug >Reporter: Sidharth Kumar Mishra >Priority: Major > Attachments: ScreenShot HookContext.png, ScreenShot > RunPostExecHook.png, ScreenShot runBeforeExecution.png > > > Problem: When I run the below mentioned queries, the last query should have > given the proper hive lineage info (through HookContext) from table_b to > table_t. > * Create table table_t (id int) partitioned by (dob date); > * Create table table_b (id int) partitioned by (dob date); > * from table_b a insert overwrite table table_t select a.id,a.dob; > Note : for CTAS query from a partitioned table , this issue is not seen. Only > for insert queries like insert into select * from and query > like above, issue is seen. > This issue is seen in latest HDP builds as well. > > Technical Observations: > At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through > hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot > from IntelliJ. > !ScreenShot RunPostExecHook.png|width=2726,height=1599! > > I found that the PrivateHookContext is getting created with proper outputs > value as shown below initially: > !ScreenShot HookContext.png! > The same is passed properly to runBeforeExecutionHook as shown below: > !ScreenShot runBeforeExecution.png! > > Later when we pass HookContext to runPostExecHooks, there is no output > populated. Kindly check the reason and let me know if you need any further > information from my end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324749
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332270501
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionService.java
##
@@ -0,0 +1,159 @@
+package org.apache.hadoop.hive.ql.schq;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.ql.DriverFactory;
+import org.apache.hadoop.hive.ql.IDriver;
+import org.apache.hadoop.hive.ql.processors.CommandProcessorResponse;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class ScheduledQueryExecutionService implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryExecutionService.class);
+
+ private ScheduledQueryExecutionContext context;
+ private ScheduledQueryExecutor worker;
+
+ public static ScheduledQueryExecutionService
startScheduledQueryExecutorService(HiveConf conf) {
+MetastoreBasedScheduledQueryService qService = new
MetastoreBasedScheduledQueryService(conf);
+ExecutorService executor =
+Executors.newCachedThreadPool(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Scheduled Query Thread
%d").build());
+ScheduledQueryExecutionContext ctx = new
ScheduledQueryExecutionContext(executor, conf, qService);
+return new ScheduledQueryExecutionService(ctx);
+ }
+
+ public ScheduledQueryExecutionService(ScheduledQueryExecutionContext ctx) {
+context = ctx;
+ctx.executor.submit(worker = new ScheduledQueryExecutor());
+ctx.executor.submit(new ProgressReporter());
+ }
+
+ class ScheduledQueryExecutor implements Runnable {
+
+private ScheduledQueryProgressInfo info;
+
+@Override
+public void run() {
+ while (true) {
+ScheduledQueryPollResponse q =
context.schedulerService.scheduledQueryPoll();
+if (q.isSetExecutionId()) {
+ try{
+processQuery(q);
+ } catch (Throwable t) {
+LOG.error("Unexpected exception during scheduled query
processing", t);
+ }
+} else {
+ try {
+Thread.sleep(context.getIdleSleepTime());
+ } catch (InterruptedException e) {
+Thread.currentThread().interrupt();
+LOG.info("interrupted");
+break;
+ }
+}
+ }
+}
+
+QueryState lastState;
+public synchronized void reportQueryProgress() {
+ if (info != null) {
+if(lastState == QueryState.FINISHED || lastState ==
QueryState.ERRORED) {
+ return;
+}
+LOG.info("Reporting query progress of {} as {} err:{}",
info.getScheduledExecutionId(), info.getState(),
+info.getErrorMessage());
+context.schedulerService.scheduledQueryProgress(info);
+lastState=info.getState();
+ }
+}
+
+private void processQuery(ScheduledQueryPollResponse q) {
+ SessionState state = null;
+ try {
+state = SessionState.start(context.conf);
+info = new ScheduledQueryProgressInfo();
+info.setScheduledExecutionId(q.getExecutionId());
+info.setState(QueryState.EXECUTING);
+reportQueryProgress();
+try (
+ IDriver driver =
DriverFactory.newDriver(DriverFactory.getNewQueryState(context.conf),
q.getUser(), null)) {
+ info.setExecutorQueryId(driver.getQueryState().getQueryId());
+ CommandProcessorResponse resp;
+ resp = driver.run(q.getQuery());
+ if (resp.getResponseCode() != 0) {
+throw resp;
+ }
+ info.setState(QueryState.FINISHED);
+}
+ } catch (Throwable t) {
+info.setErrorMessage(getErrorStringForException(t));
+info.setState(QueryState.ERRORED);
+ } finally {
+if (state != null) {
+ try {
+state.close();
+ } catch (Throwable e) {
+ }
+}
+
+synchronized (this) {
Review comment:
Why do we
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324704 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r331849457 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/IScheduledQueryMaintenanceService.java ## @@ -0,0 +1,19 @@ +package org.apache.hadoop.hive.ql.schq; Review comment: Do you think it is better to make the package name self-explanatory? e.g. `org.apache.hadoop.hive.ql.scheduled.*`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324704) Time Spent: 50m (was: 40m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 50m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324712
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331670949
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestRestrictedList.java
##
@@ -64,7 +64,9 @@ public static void startServices() throws Exception {
hiveConf.setBoolVar(ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
miniHS2 = new
MiniHS2.Builder().withMiniMR().withRemoteMetastore().withConf(hiveConf).build();
-miniHS2.start(new HashMap());
+HashMap confOverlay = new HashMap<>();
Review comment:
`HashMap` -> `Map`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324712)
Time Spent: 1.5h (was: 1h 20m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324713 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332261373 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionService.java ## @@ -0,0 +1,159 @@ +package org.apache.hadoop.hive.ql.schq; Review comment: Missing license. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324713) Time Spent: 1.5h (was: 1h 20m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 1.5h > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324705 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332149917 ## File path: ql/src/java/org/apache/hadoop/hive/ql/hooks/ScheduledQueryCreationRegistryHook.java ## @@ -0,0 +1,38 @@ +package org.apache.hadoop.hive.ql.hooks; + +import java.util.HashSet; +import java.util.Set; + +import org.apache.hadoop.hive.ql.parse.ASTNode; +import org.apache.hadoop.hive.ql.parse.AbstractSemanticAnalyzerHook; +import org.apache.hadoop.hive.ql.parse.HiveParser; +import org.apache.hadoop.hive.ql.parse.HiveSemanticAnalyzerHookContext; +import org.apache.hadoop.hive.ql.parse.SemanticException; + +/** + * This hook helps to do some cleanup after tests are creating scheduled queries. Review comment: Is this only for tests? Should we move it to tests/itests subdirectory then? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324705) Time Spent: 1h (was: 50m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 1h > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324721
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332280503
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MScheduledExecution.java
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore.model;
+
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+
+public class MScheduledExecution {
+
+ public MScheduledExecution() {
+ }
+
+ private int scheduledExecutionId;
+ private MScheduledQuery scheduledQuery;
+ private String executorQueryId;
+ private String state;
+ private Integer startTime;
+ private Integer endTime;
+ private String errorMessage;
+ private Integer lastUpdateTime;
+
+ @Override
+ public String toString() {
+return String.format("state: %s, scheduledQuery: %s, execId: %d", state,
scheduledQuery.getScheduleName(),
+executorQueryId);
+ }
+
+ // public MScheduledQuery2(ScheduledQuery s) {
Review comment:
We can remove commented code?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324721)
Time Spent: 1h 50m (was: 1h 40m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324728
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332255167
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ScheduledQueryAnalyzer.java
##
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import static org.apache.hadoop.hive.ql.parse.HiveParser.TOK_IFEXISTS;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.Stack;
+
+import org.antlr.runtime.tree.Tree;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery._Fields;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
+import org.apache.hadoop.hive.ql.exec.FunctionUtils;
+import org.apache.hadoop.hive.ql.exec.TaskFactory;
+import org.apache.hadoop.hive.ql.hooks.WriteEntity;
+import org.apache.hadoop.hive.ql.lib.Dispatcher;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.PreOrderWalker;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.HiveOperation;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ScheduledQueryAnalyzer extends BaseSemanticAnalyzer {
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryAnalyzer.class);
+
+ public ScheduledQueryAnalyzer(QueryState queryState) throws
SemanticException {
+super(queryState);
+ }
+
+ @Override
+ public void analyzeInternal(ASTNode ast) throws SemanticException {
+ScheduledQueryMaintWork work;
+ScheduledQueryMaintenanceRequestType type =
translateAstType(ast.getToken().getType());
+ScheduledQuery schq = interpretAstNode(ast);
+fillScheduledQuery(type, schq);
+LOG.info("scheduled query operation: " + type + " " + schq);
+try {
+ schq.validate();
+} catch (TException e) {
+ throw new SemanticException("ScheduledQuery is invalid", e);
+}
+work = new ScheduledQueryMaintWork(type, schq);
+rootTasks.add(TaskFactory.get(work));
+ }
+
+ private void fillScheduledQuery(ScheduledQueryMaintenanceRequestType type,
ScheduledQuery schq)
+ throws SemanticException {
+if (type == ScheduledQueryMaintenanceRequestType.INSERT) {
+ populateUnfilled(schq, buildEmptySchq());
+} else {
+ try {
+ScheduledQuery oldSchq =
db.getMSC().getScheduledQuery(schq.getScheduleKey());
+populateUnfilled(schq, oldSchq);
+ } catch (TException e) {
+throw new SemanticException("unable to get Scheduled query" + e);
+ }
+}
+ }
+
+ private ScheduledQuery buildEmptySchq() {
+ScheduledQuery ret = new ScheduledQuery();
+// ret.setScheduleKey() -- not populated
+// ret.setSchedule(schedule);
+// ret.setQuery(query);
+ret.setEnabled(true);
+
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324734
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332273651
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
##
@@ -363,6 +363,10 @@ public HivePrivilegeObjectType getObjectType() {
op2Priv.put(HiveOperationType.LOCKTABLE,
PrivRequirement.newIOPrivRequirement(null, null));
op2Priv.put(HiveOperationType.UNLOCKTABLE,
PrivRequirement.newIOPrivRequirement(null, null));
+op2Priv.put(HiveOperationType.CREATE_SCHEDULED_QUERY,
PrivRequirement.newIOPrivRequirement(null, ADMIN_PRIV_AR));
Review comment:
Why is first parameter _null_?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324734)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324740
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331850345
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintTask.java
##
@@ -0,0 +1,46 @@
+package org.apache.hadoop.hive.ql.exec.schq;
+
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequest;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.thrift.TException;
+
+public class ScheduledQueryMaintTask extends Task {
+
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public String getName() {
+return "SCHEDULED QUERY MAINT TASK";
Review comment:
`MAINTENANCE`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324740)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324720
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332280312
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ScheduledQueryExecutionsMaintTask.java
##
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hive.metastore.RawStore;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Metastore task to remove old scheduled query executions.
+ */
+public class ScheduledQueryExecutionsMaintTask implements MetastoreTaskThread {
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryExecutionsMaintTask.class);
+
+ private Configuration conf;
+
+ @Override
+ public long runFrequency(TimeUnit unit) {
+return MetastoreConf.getTimeVar(conf,
MetastoreConf.ConfVars.SCHEDULED_QUERIES_EXECUTION_MAINT_TASK_FREQUENCY,
+unit);
+ }
+
+ @Override
+ public void setConf(Configuration configuration) {
+conf = configuration;
+ }
+
+ @Override
+ public Configuration getConf() {
+return conf;
+ }
+
+ @Override
+ public void run() {
+
Review comment:
Knitpick. Remove new lines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324720)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324716
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332256244
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ScheduledQueryAnalyzer.java
##
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import static org.apache.hadoop.hive.ql.parse.HiveParser.TOK_IFEXISTS;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.Stack;
+
+import org.antlr.runtime.tree.Tree;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery._Fields;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
+import org.apache.hadoop.hive.ql.exec.FunctionUtils;
+import org.apache.hadoop.hive.ql.exec.TaskFactory;
+import org.apache.hadoop.hive.ql.hooks.WriteEntity;
+import org.apache.hadoop.hive.ql.lib.Dispatcher;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.PreOrderWalker;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.HiveOperation;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ScheduledQueryAnalyzer extends BaseSemanticAnalyzer {
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryAnalyzer.class);
+
+ public ScheduledQueryAnalyzer(QueryState queryState) throws
SemanticException {
+super(queryState);
+ }
+
+ @Override
+ public void analyzeInternal(ASTNode ast) throws SemanticException {
+ScheduledQueryMaintWork work;
+ScheduledQueryMaintenanceRequestType type =
translateAstType(ast.getToken().getType());
+ScheduledQuery schq = interpretAstNode(ast);
+fillScheduledQuery(type, schq);
+LOG.info("scheduled query operation: " + type + " " + schq);
+try {
+ schq.validate();
+} catch (TException e) {
+ throw new SemanticException("ScheduledQuery is invalid", e);
+}
+work = new ScheduledQueryMaintWork(type, schq);
+rootTasks.add(TaskFactory.get(work));
+ }
+
+ private void fillScheduledQuery(ScheduledQueryMaintenanceRequestType type,
ScheduledQuery schq)
+ throws SemanticException {
+if (type == ScheduledQueryMaintenanceRequestType.INSERT) {
+ populateUnfilled(schq, buildEmptySchq());
+} else {
+ try {
+ScheduledQuery oldSchq =
db.getMSC().getScheduledQuery(schq.getScheduleKey());
+populateUnfilled(schq, oldSchq);
+ } catch (TException e) {
+throw new SemanticException("unable to get Scheduled query" + e);
+ }
+}
+ }
+
+ private ScheduledQuery buildEmptySchq() {
+ScheduledQuery ret = new ScheduledQuery();
+// ret.setScheduleKey() -- not populated
+// ret.setSchedule(schedule);
+// ret.setQuery(query);
+ret.setEnabled(true);
+
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324725
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332263137
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##
@@ -667,6 +667,19 @@ public static ConfVars getMetaConf(String name) {
RUNTIME_STATS_MAX_AGE("runtime.stats.max.age",
"hive.metastore.runtime.stats.max.age", 86400 * 3, TimeUnit.SECONDS,
"Stat entries which are older than this are removed."),
+SCHEDULED_QUERIES_CRON_SYNTAX("scheduled.queries.cron.syntax",
"hive.metastore.scheduled.queries.cron.syntax",
+"QUARTZ", new StringSetValidator("UNIX", "QUARTZ", "CRON4J", "SPRING"),
+"Defines the format of the schedule expressions to be used in
scheduled queries."),
+
SCHEDULED_QUERIES_EXECUTION_PROGRESS_TIMEOUT("scheduled.queries.execution.timeout",
+"hive.metastore.scheduled.queries.progress.timeout", 120,
TimeUnit.SECONDS,
+"If a scheduled query is not making progress for this amount of time
it will be considered TIMED_OUT"),
+
SCHEDULED_QUERIES_EXECUTION_MAINT_TASK_FREQUENCY("scheduled.queries.execution.maint.task.frequency",
+"hive.metastore.scheduled.queries.execution.clean.frequency", 60,
TimeUnit.SECONDS,
+"Interval of scheduled query maintenance task. Which removes
executions above max age; and marks executions as timed out if the condition is
met"),
+SCHEDULED_QUERIES_EXECUTION_MAX_AGE("scheduled.queries.execution.max.age",
+"hive.metastore.scheduled.queries.execution.max.age", 3 * 86400,
TimeUnit.SECONDS,
Review comment:
This is three days. I think we should set it longer since sometimes it may
take more than three days to dig into a problem... We can set this to one month
by default? If there are too many events, maybe we can have two different
intervals, one for deleting `EXECUTING` events (more often) and one of the rest
(less often)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324725)
Time Spent: 2h 10m (was: 2h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324745 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332282027 ## File path: standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql ## @@ -18,6 +18,36 @@ UPDATE "APP"."TAB_COL_STATS" SET ENGINE = 'hive' WHERE ENGINE IS NULL; ALTER TABLE "APP"."PART_COL_STATS" ADD ENGINE VARCHAR(128); UPDATE "APP"."PART_COL_STATS" SET ENGINE = 'hive' WHERE ENGINE IS NULL; +CREATE TABLE "APP"."SCHEDULED_QUERIES" ( Review comment: I am wondering whether you have tested all the upgrade scripts for the different RDBMS, e.g., with your docker image or the one that Alan created? I believe so but just to confirm with you, I know it is easy to have problems in some of the upgrade scripts that remain unnoticed for a while :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324745) Time Spent: 2h 50m (was: 2h 40m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324733
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332252082
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ScheduledQueryAnalyzer.java
##
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import static org.apache.hadoop.hive.ql.parse.HiveParser.TOK_IFEXISTS;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.Stack;
+
+import org.antlr.runtime.tree.Tree;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery._Fields;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
+import org.apache.hadoop.hive.ql.exec.FunctionUtils;
+import org.apache.hadoop.hive.ql.exec.TaskFactory;
+import org.apache.hadoop.hive.ql.hooks.WriteEntity;
+import org.apache.hadoop.hive.ql.lib.Dispatcher;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.PreOrderWalker;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.HiveOperation;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ScheduledQueryAnalyzer extends BaseSemanticAnalyzer {
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryAnalyzer.class);
+
+ public ScheduledQueryAnalyzer(QueryState queryState) throws
SemanticException {
+super(queryState);
+ }
+
+ @Override
+ public void analyzeInternal(ASTNode ast) throws SemanticException {
+ScheduledQueryMaintWork work;
+ScheduledQueryMaintenanceRequestType type =
translateAstType(ast.getToken().getType());
+ScheduledQuery schq = interpretAstNode(ast);
+fillScheduledQuery(type, schq);
+LOG.info("scheduled query operation: " + type + " " + schq);
+try {
+ schq.validate();
+} catch (TException e) {
+ throw new SemanticException("ScheduledQuery is invalid", e);
+}
+work = new ScheduledQueryMaintWork(type, schq);
+rootTasks.add(TaskFactory.get(work));
+ }
+
+ private void fillScheduledQuery(ScheduledQueryMaintenanceRequestType type,
ScheduledQuery schq)
+ throws SemanticException {
+if (type == ScheduledQueryMaintenanceRequestType.INSERT) {
+ populateUnfilled(schq, buildEmptySchq());
+} else {
+ try {
+ScheduledQuery oldSchq =
db.getMSC().getScheduledQuery(schq.getScheduleKey());
+populateUnfilled(schq, oldSchq);
Review comment:
Should this logic work in the opposite way? Updating old scheduled query
with new values? I am wondering if we end up with duplicate queries otherwise.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324723
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332237943
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ScheduledQueryAnalyzer.java
##
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import static org.apache.hadoop.hive.ql.parse.HiveParser.TOK_IFEXISTS;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.Stack;
+
+import org.antlr.runtime.tree.Tree;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery._Fields;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
+import org.apache.hadoop.hive.ql.exec.FunctionUtils;
+import org.apache.hadoop.hive.ql.exec.TaskFactory;
+import org.apache.hadoop.hive.ql.hooks.WriteEntity;
+import org.apache.hadoop.hive.ql.lib.Dispatcher;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.PreOrderWalker;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.HiveOperation;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ScheduledQueryAnalyzer extends BaseSemanticAnalyzer {
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryAnalyzer.class);
+
+ public ScheduledQueryAnalyzer(QueryState queryState) throws
SemanticException {
+super(queryState);
+ }
+
+ @Override
+ public void analyzeInternal(ASTNode ast) throws SemanticException {
+ScheduledQueryMaintWork work;
+ScheduledQueryMaintenanceRequestType type =
translateAstType(ast.getToken().getType());
+ScheduledQuery schq = interpretAstNode(ast);
+fillScheduledQuery(type, schq);
+LOG.info("scheduled query operation: " + type + " " + schq);
+try {
+ schq.validate();
+} catch (TException e) {
+ throw new SemanticException("ScheduledQuery is invalid", e);
+}
+work = new ScheduledQueryMaintWork(type, schq);
+rootTasks.add(TaskFactory.get(work));
+ }
+
+ private void fillScheduledQuery(ScheduledQueryMaintenanceRequestType type,
ScheduledQuery schq)
+ throws SemanticException {
+if (type == ScheduledQueryMaintenanceRequestType.INSERT) {
+ populateUnfilled(schq, buildEmptySchq());
+} else {
+ try {
+ScheduledQuery oldSchq =
db.getMSC().getScheduledQuery(schq.getScheduleKey());
+populateUnfilled(schq, oldSchq);
+ } catch (TException e) {
+throw new SemanticException("unable to get Scheduled query" + e);
+ }
+}
+ }
+
+ private ScheduledQuery buildEmptySchq() {
+ScheduledQuery ret = new ScheduledQuery();
+// ret.setScheduleKey() -- not populated
+// ret.setSchedule(schedule);
+// ret.setQuery(query);
+ret.setEnabled(true);
+
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324748
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331850284
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintTask.java
##
@@ -0,0 +1,46 @@
+package org.apache.hadoop.hive.ql.exec.schq;
+
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequest;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.thrift.TException;
+
+public class ScheduledQueryMaintTask extends Task {
Review comment:
Should we use full name to make it more readable? I.e. `Maintenance`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324748)
Time Spent: 2h 50m (was: 2h 40m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324754 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r331849694 ## File path: ql/src/java/org/apache/hadoop/hive/ql/Context.java ## @@ -170,6 +170,7 @@ // Load data rewrite private Table tempTableForLoad; + private boolean enableUnparse; Review comment: Please add to this variable, i.e., how and where it is used. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324754) Time Spent: 3h (was: 2h 50m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 3h > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324731 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332273015 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryMaintWork.java ## @@ -0,0 +1,27 @@ +package org.apache.hadoop.hive.ql.schq; Review comment: Missing license. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324731) Time Spent: 2h 20m (was: 2h 10m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 20m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324746
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332283837
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestMetastoreScheduledQueries.java
##
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.client;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import javax.jdo.JDOObjectNotFoundException;
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.ObjectStore;
+import org.apache.hadoop.hive.metastore.ObjectStoreTestHook;
+import org.apache.hadoop.hive.metastore.PersistenceManagerProvider;
+import org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.InvalidInputException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequest;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollRequest;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.metastore.minihms.AbstractMetaStoreService;
+import org.apache.hadoop.hive.metastore.model.MScheduledExecution;
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+@Category(MetastoreUnitTest.class)
+public class TestMetastoreScheduledQueries extends MetaStoreClientTest {
+ private final AbstractMetaStoreService metaStore;
+ private IMetaStoreClient client;
+
+ public TestMetastoreScheduledQueries(String name, AbstractMetaStoreService
metaStore) throws Exception {
+metaStore.getConf().set("scheduled.queries.cron.syntax", "QUARTZ");
+metaStore.getConf().set("scheduled.queries.progress.timeout", "3");
+this.metaStore = metaStore;
+ }
+
+ @Before
+ public void setUp() throws Exception {
+client = metaStore.getClient();
+
+ }
+
+ @After
+ public void tearDown() throws Exception {
+try {
+ client.close();
+} catch (Exception e) {
+ // HIVE-19729: Shallow the exceptions based on the discussion in the Jira
Review comment:
Not sure about that specific issue HIVE-19729... But should we at least
print them at WARN level or appended to another message?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324746)
Time Spent: 2h 50m (was: 2h
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324732
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332280126
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -12583,4 +12600,306 @@ public static boolean isCurrentStatsValidForTheQuery(
return false;
}
+
+ @Override
+ public ScheduledQueryPollResponse
scheduledQueryPoll(ScheduledQueryPollRequest request) {
+String namespace = request.getClusterNamespace();
+boolean commited = false;
+ScheduledQueryPollResponse ret = new ScheduledQueryPollResponse();
+try {
+ openTransaction();
+ Query q = pm.newQuery(MScheduledQuery.class,
+ "nextExecution <= now && enabled && clusterNamespace == ns");
+ q.setSerializeRead(true);
+ q.declareParameters("java.lang.Integer now, java.lang.String ns");
+ q.setOrdering("nextExecution");
+ int now = (int) (System.currentTimeMillis() / 1000);
+ List results = (List) q.execute(now,
request.getClusterNamespace());
+ if (results == null || results.isEmpty()) {
+return new ScheduledQueryPollResponse();
+ }
+ MScheduledQuery schq = results.get(0);
+ Integer plannedExecutionTime = schq.getNextExecution();
+ schq.setNextExecution(computeNextExecutionTime(schq.getSchedule()));
+
+ MScheduledExecution execution = new MScheduledExecution();
+ execution.setScheduledQuery(schq);
+ execution.setState(QueryState.INITED);
+ execution.setStartTime(now);
+ execution.setLastUpdateTime(now);
+ pm.makePersistent(execution);
+ pm.makePersistent(schq);
+ ObjectStoreTestHook.onScheduledQueryPoll();
+ commited = commitTransaction();
+ ret.setScheduleKey(schq.getScheduleKey());
+ ret.setQuery(schq.getQuery());
+ ret.setUser(schq.getUser());
+ int executionId = ((IntIdentity) pm.getObjectId(execution)).getKey();
+ ret.setExecutionId(executionId);
+} catch (JDOException e) {
+ LOG.debug("Caught jdo exception; exclusive", e);
+ commited = false;
+} finally {
+ if (commited) {
+return ret;
+ } else {
+rollbackTransaction();
+return new ScheduledQueryPollResponse();
+ }
+}
+ }
+
+ @Override
+ public void scheduledQueryProgress(ScheduledQueryProgressInfo info) throws
InvalidOperationException {
+boolean commited = false;
+try {
+ openTransaction();
+ MScheduledExecution execution =
pm.getObjectById(MScheduledExecution.class, info.getScheduledExecutionId());
+ if (!validateStateChange(execution.getState(), info.getState())) {
+throw new InvalidOperationException("Invalid state change: " +
execution.getState() + "=>" + info.getState());
+ }
+ execution.setState(info.getState());
+ if (info.isSetExecutorQueryId()) {
+execution.setExecutorQueryId(info.getExecutorQueryId());
+ }
+ if (info.isSetErrorMessage()) {
+execution.setErrorMessage(info.getErrorMessage());
+ }
+
+ switch (info.getState()) {
+ case INITED:
+ case EXECUTING:
+execution.setLastUpdateTime((int) (System.currentTimeMillis() / 1000));
+break;
+ case ERRORED:
+ case FINISHED:
+ case TIMED_OUT:
+execution.setEndTime((int) (System.currentTimeMillis() / 1000));
+execution.setLastUpdateTime(null);
+ }
+ pm.makePersistent(execution);
+ commited = commitTransaction();
+} finally {
+ if (commited) {
+ } else {
+rollbackTransaction();
+ }
+}
+ }
+
+ private boolean validateStateChange(QueryState from, QueryState to) {
+switch (from) {
+case INITED:
+ return to != QueryState.INITED;
+case EXECUTING:
+ return to == QueryState.FINISHED
+ || to == QueryState.EXECUTING
+ || to == QueryState.ERRORED;
+default:
+ return false;
+}
+ }
+
+ private Integer computeNextExecutionTime(String schedule) throws
InvalidInputException {
+CronType cronType = CronType.valueOf(MetastoreConf.getVar(this.conf,
ConfVars.SCHEDULED_QUERIES_CRON_SYNTAX));
+
+CronDefinition cronDefinition =
CronDefinitionBuilder.instanceDefinitionFor(cronType);
+CronParser parser = new CronParser(cronDefinition);
+
+// Get date for last execution
+try {
+ ZonedDateTime now = ZonedDateTime.now();
+ ExecutionTime executionTime =
ExecutionTime.forCron(parser.parse(schedule));
+ Optional nextExecution =
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324711
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332150024
##
File path: ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
##
@@ -310,7 +310,7 @@ private void verifyState(QueryPlan queryPlan) throws
LockException {
" for " + getQueryIdWaterMark(queryPlan));
}
if(queryPlan.getOperation() == null) {
- throw new IllegalStateException("Unkown HiverOperation for " +
getQueryIdWaterMark(queryPlan));
+ throw new IllegalStateException("Unkown HiveOperation(null) for " +
getQueryIdWaterMark(queryPlan));
Review comment:
Typo: `Unkown` -> `Unknown`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324711)
Time Spent: 1.5h (was: 1h 20m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324709
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332269978
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionService.java
##
@@ -0,0 +1,159 @@
+package org.apache.hadoop.hive.ql.schq;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.ql.DriverFactory;
+import org.apache.hadoop.hive.ql.IDriver;
+import org.apache.hadoop.hive.ql.processors.CommandProcessorResponse;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class ScheduledQueryExecutionService implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryExecutionService.class);
+
+ private ScheduledQueryExecutionContext context;
+ private ScheduledQueryExecutor worker;
+
+ public static ScheduledQueryExecutionService
startScheduledQueryExecutorService(HiveConf conf) {
+MetastoreBasedScheduledQueryService qService = new
MetastoreBasedScheduledQueryService(conf);
+ExecutorService executor =
+Executors.newCachedThreadPool(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Scheduled Query Thread
%d").build());
+ScheduledQueryExecutionContext ctx = new
ScheduledQueryExecutionContext(executor, conf, qService);
+return new ScheduledQueryExecutionService(ctx);
+ }
+
+ public ScheduledQueryExecutionService(ScheduledQueryExecutionContext ctx) {
+context = ctx;
+ctx.executor.submit(worker = new ScheduledQueryExecutor());
+ctx.executor.submit(new ProgressReporter());
+ }
+
+ class ScheduledQueryExecutor implements Runnable {
+
+private ScheduledQueryProgressInfo info;
+
+@Override
+public void run() {
+ while (true) {
+ScheduledQueryPollResponse q =
context.schedulerService.scheduledQueryPoll();
+if (q.isSetExecutionId()) {
+ try{
+processQuery(q);
+ } catch (Throwable t) {
+LOG.error("Unexpected exception during scheduled query
processing", t);
+ }
+} else {
+ try {
+Thread.sleep(context.getIdleSleepTime());
+ } catch (InterruptedException e) {
+Thread.currentThread().interrupt();
+LOG.info("interrupted");
+break;
+ }
+}
+ }
+}
+
+QueryState lastState;
+public synchronized void reportQueryProgress() {
+ if (info != null) {
+if(lastState == QueryState.FINISHED || lastState ==
QueryState.ERRORED) {
+ return;
+}
+LOG.info("Reporting query progress of {} as {} err:{}",
info.getScheduledExecutionId(), info.getState(),
+info.getErrorMessage());
+context.schedulerService.scheduledQueryProgress(info);
+lastState=info.getState();
+ }
+}
+
+private void processQuery(ScheduledQueryPollResponse q) {
+ SessionState state = null;
+ try {
+state = SessionState.start(context.conf);
+info = new ScheduledQueryProgressInfo();
+info.setScheduledExecutionId(q.getExecutionId());
+info.setState(QueryState.EXECUTING);
+reportQueryProgress();
+try (
+ IDriver driver =
DriverFactory.newDriver(DriverFactory.getNewQueryState(context.conf),
q.getUser(), null)) {
+ info.setExecutorQueryId(driver.getQueryState().getQueryId());
Review comment:
This is cool... Basically scheduled execution id matches the query execution
id generated by HS2... which makes it easy to map a scheduled query to the
corresponding execution in YARN. Right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324729
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332279975
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -12583,4 +12600,306 @@ public static boolean isCurrentStatsValidForTheQuery(
return false;
}
+
+ @Override
+ public ScheduledQueryPollResponse
scheduledQueryPoll(ScheduledQueryPollRequest request) {
+String namespace = request.getClusterNamespace();
+boolean commited = false;
+ScheduledQueryPollResponse ret = new ScheduledQueryPollResponse();
+try {
+ openTransaction();
+ Query q = pm.newQuery(MScheduledQuery.class,
+ "nextExecution <= now && enabled && clusterNamespace == ns");
+ q.setSerializeRead(true);
+ q.declareParameters("java.lang.Integer now, java.lang.String ns");
+ q.setOrdering("nextExecution");
+ int now = (int) (System.currentTimeMillis() / 1000);
+ List results = (List) q.execute(now,
request.getClusterNamespace());
+ if (results == null || results.isEmpty()) {
+return new ScheduledQueryPollResponse();
+ }
+ MScheduledQuery schq = results.get(0);
+ Integer plannedExecutionTime = schq.getNextExecution();
+ schq.setNextExecution(computeNextExecutionTime(schq.getSchedule()));
+
+ MScheduledExecution execution = new MScheduledExecution();
+ execution.setScheduledQuery(schq);
+ execution.setState(QueryState.INITED);
+ execution.setStartTime(now);
+ execution.setLastUpdateTime(now);
+ pm.makePersistent(execution);
+ pm.makePersistent(schq);
+ ObjectStoreTestHook.onScheduledQueryPoll();
+ commited = commitTransaction();
+ ret.setScheduleKey(schq.getScheduleKey());
+ ret.setQuery(schq.getQuery());
+ ret.setUser(schq.getUser());
+ int executionId = ((IntIdentity) pm.getObjectId(execution)).getKey();
+ ret.setExecutionId(executionId);
+} catch (JDOException e) {
+ LOG.debug("Caught jdo exception; exclusive", e);
+ commited = false;
+} finally {
+ if (commited) {
+return ret;
+ } else {
+rollbackTransaction();
+return new ScheduledQueryPollResponse();
+ }
+}
+ }
+
+ @Override
+ public void scheduledQueryProgress(ScheduledQueryProgressInfo info) throws
InvalidOperationException {
+boolean commited = false;
+try {
+ openTransaction();
+ MScheduledExecution execution =
pm.getObjectById(MScheduledExecution.class, info.getScheduledExecutionId());
+ if (!validateStateChange(execution.getState(), info.getState())) {
+throw new InvalidOperationException("Invalid state change: " +
execution.getState() + "=>" + info.getState());
Review comment:
Cool!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324729)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324735
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332283611
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestMetastoreScheduledQueries.java
##
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.client;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import javax.jdo.JDOObjectNotFoundException;
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.ObjectStore;
+import org.apache.hadoop.hive.metastore.ObjectStoreTestHook;
+import org.apache.hadoop.hive.metastore.PersistenceManagerProvider;
+import org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.InvalidInputException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequest;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollRequest;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.metastore.minihms.AbstractMetaStoreService;
+import org.apache.hadoop.hive.metastore.model.MScheduledExecution;
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+@Category(MetastoreUnitTest.class)
+public class TestMetastoreScheduledQueries extends MetaStoreClientTest {
+ private final AbstractMetaStoreService metaStore;
+ private IMetaStoreClient client;
+
+ public TestMetastoreScheduledQueries(String name, AbstractMetaStoreService
metaStore) throws Exception {
+metaStore.getConf().set("scheduled.queries.cron.syntax", "QUARTZ");
+metaStore.getConf().set("scheduled.queries.progress.timeout", "3");
+this.metaStore = metaStore;
+ }
+
+ @Before
+ public void setUp() throws Exception {
+client = metaStore.getClient();
+
+ }
+
+ @After
+ public void tearDown() throws Exception {
+try {
+ client.close();
+} catch (Exception e) {
+ // HIVE-19729: Shallow the exceptions based on the discussion in the Jira
+}
+client = null;
+ }
+
+ @Test(expected = NoSuchObjectException.class)
+ public void testNonExistent() throws Exception {
+client.getScheduledQuery(new ScheduledQueryKey("nonExistent", "x"));
+ }
+
+ @Test
+ public void testCreate() throws Exception {
+
Review comment:
Knitpick. New line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service,
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324718 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r331850110 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintTask.java ## @@ -0,0 +1,46 @@ +package org.apache.hadoop.hive.ql.exec.schq; Review comment: License missing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324718) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324747
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332274057
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
##
@@ -2066,6 +2074,71 @@ dropMaterializedViewStatement
: KW_DROP KW_MATERIALIZED KW_VIEW ifExists? viewName ->
^(TOK_DROP_MATERIALIZED_VIEW viewName ifExists?)
;
+createScheduledQueryStatement
+@init { pushMsg("create scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_CREATE KW_SCHEDULED KW_QUERY name=identifier
+scheduleSpec
+executedAsSpec?
+enableSpecification?
+definedAsSpec
+-> ^(TOK_CREATE_SCHEDULED_QUERY
+$name
+scheduleSpec
+executedAsSpec?
+enableSpecification?
+definedAsSpec
+)
+;
+
+dropScheduledQueryStatement
+@init { pushMsg("drop scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_DROP KW_SCHEDULED KW_QUERY name=identifier
+-> ^(TOK_DROP_SCHEDULED_QUERY
+$name
+)
+;
+
+
+alterScheduledQueryStatement
+@init { pushMsg("alter scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_ALTER KW_SCHEDULED KW_QUERY name=identifier
+mod=alterScheduledQueryChange
+-> ^(TOK_ALTER_SCHEDULED_QUERY
+$name
+$mod
+)
+;
+
+alterScheduledQueryChange
+@init { pushMsg("alter scheduled query change", state); }
+@after { popMsg(state); }
+: scheduleSpec
+| executedAsSpec
+| enableSpecification
+| definedAsSpec
+;
+
+scheduleSpec
+@init { pushMsg("schedule specification", state); }
+@after { popMsg(state); }
+: KW_CRON cronString=StringLiteral -> ^(TOK_CRON $cronString)
+;
+
+executedAsSpec
+@init { pushMsg("executedAs specification", state); }
+@after { popMsg(state); }
+: KW_EXECUTED KW_AS executedAs=StringLiteral -> ^(TOK_EXECUTED_AS
$executedAs)
+;
+
+definedAsSpec
+@init { pushMsg("definedAs specification", state); }
+@after { popMsg(state); }
+: KW_DEFINED? KW_AS statement -> ^(TOK_QUERY statement)
Review comment:
I think we can remove `DEFINED`, since it is redundant and we do not use it
for most statements.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324747)
Time Spent: 2h 50m (was: 2h 40m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324736
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332255906
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
##
@@ -2318,4 +2318,10 @@ public void addPropertyReadEntry(Map
tblProps, Set i
}
}
}
+
+ protected void unparse() {
+UnparseTranslator unparseTranslator = new UnparseTranslator(conf);
Review comment:
What does it do? What is the difference with the overriden method in
`SemanticAnalyzer`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324736)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324751
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332282330
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestMetastoreScheduledQueries.java
##
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.client;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import javax.jdo.JDOObjectNotFoundException;
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.ObjectStore;
+import org.apache.hadoop.hive.metastore.ObjectStoreTestHook;
+import org.apache.hadoop.hive.metastore.PersistenceManagerProvider;
+import org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.InvalidInputException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequest;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollRequest;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.metastore.minihms.AbstractMetaStoreService;
+import org.apache.hadoop.hive.metastore.model.MScheduledExecution;
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+@Category(MetastoreUnitTest.class)
+public class TestMetastoreScheduledQueries extends MetaStoreClientTest {
+ private final AbstractMetaStoreService metaStore;
+ private IMetaStoreClient client;
+
+ public TestMetastoreScheduledQueries(String name, AbstractMetaStoreService
metaStore) throws Exception {
+metaStore.getConf().set("scheduled.queries.cron.syntax", "QUARTZ");
+metaStore.getConf().set("scheduled.queries.progress.timeout", "3");
+this.metaStore = metaStore;
+ }
+
+ @Before
+ public void setUp() throws Exception {
+client = metaStore.getClient();
+
Review comment:
Knitpick. New line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324751)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324741
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332157079
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
##
@@ -2066,6 +2074,71 @@ dropMaterializedViewStatement
: KW_DROP KW_MATERIALIZED KW_VIEW ifExists? viewName ->
^(TOK_DROP_MATERIALIZED_VIEW viewName ifExists?)
;
+createScheduledQueryStatement
+@init { pushMsg("create scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_CREATE KW_SCHEDULED KW_QUERY name=identifier
+scheduleSpec
+executedAsSpec?
+enableSpecification?
+definedAsSpec
+-> ^(TOK_CREATE_SCHEDULED_QUERY
+$name
+scheduleSpec
+executedAsSpec?
+enableSpecification?
+definedAsSpec
+)
+;
+
+dropScheduledQueryStatement
+@init { pushMsg("drop scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_DROP KW_SCHEDULED KW_QUERY name=identifier
+-> ^(TOK_DROP_SCHEDULED_QUERY
+$name
+)
+;
+
+
+alterScheduledQueryStatement
+@init { pushMsg("alter scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_ALTER KW_SCHEDULED KW_QUERY name=identifier
+mod=alterScheduledQueryChange
+-> ^(TOK_ALTER_SCHEDULED_QUERY
+$name
+$mod
+)
+;
+
+alterScheduledQueryChange
+@init { pushMsg("alter scheduled query change", state); }
+@after { popMsg(state); }
+: scheduleSpec
+| executedAsSpec
+| enableSpecification
+| definedAsSpec
+;
+
+scheduleSpec
+@init { pushMsg("schedule specification", state); }
+@after { popMsg(state); }
+: KW_CRON cronString=StringLiteral -> ^(TOK_CRON $cronString)
Review comment:
I think we should add some shortcuts for most commonly used schedule specs,
e.g., `DAILY AT 12.00 START IMMEDIATELY`. These could just be syntactic sugar
that translates into the corresponding cron spec, hence the changes only go at
the parser level. We can create a follow-up and discuss about it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324741)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324724
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332272923
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionService.java
##
@@ -0,0 +1,159 @@
+package org.apache.hadoop.hive.ql.schq;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.ql.DriverFactory;
+import org.apache.hadoop.hive.ql.IDriver;
+import org.apache.hadoop.hive.ql.processors.CommandProcessorResponse;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class ScheduledQueryExecutionService implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryExecutionService.class);
+
+ private ScheduledQueryExecutionContext context;
+ private ScheduledQueryExecutor worker;
+
+ public static ScheduledQueryExecutionService
startScheduledQueryExecutorService(HiveConf conf) {
+MetastoreBasedScheduledQueryService qService = new
MetastoreBasedScheduledQueryService(conf);
+ExecutorService executor =
+Executors.newCachedThreadPool(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Scheduled Query Thread
%d").build());
+ScheduledQueryExecutionContext ctx = new
ScheduledQueryExecutionContext(executor, conf, qService);
+return new ScheduledQueryExecutionService(ctx);
+ }
+
+ public ScheduledQueryExecutionService(ScheduledQueryExecutionContext ctx) {
+context = ctx;
+ctx.executor.submit(worker = new ScheduledQueryExecutor());
+ctx.executor.submit(new ProgressReporter());
+ }
+
+ class ScheduledQueryExecutor implements Runnable {
+
+private ScheduledQueryProgressInfo info;
+
+@Override
+public void run() {
+ while (true) {
+ScheduledQueryPollResponse q =
context.schedulerService.scheduledQueryPoll();
+if (q.isSetExecutionId()) {
+ try{
+processQuery(q);
+ } catch (Throwable t) {
+LOG.error("Unexpected exception during scheduled query
processing", t);
+ }
+} else {
+ try {
+Thread.sleep(context.getIdleSleepTime());
+ } catch (InterruptedException e) {
+Thread.currentThread().interrupt();
+LOG.info("interrupted");
+break;
+ }
+}
+ }
+}
+
+QueryState lastState;
+public synchronized void reportQueryProgress() {
Review comment:
Are we synchronizing this in case two requests asked for status after each
other and the first one did not return? Should we do something for the check
interval here so we do not end up with a lot of requests block here, e.g.,
exponential backoff or simply skipping query progress (and reporting it)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324724)
Time Spent: 2h 10m (was: 2h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch,
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324744 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332278060 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java ## @@ -3905,4 +3905,29 @@ void createOrDropTriggerToPoolMapping(String resourcePlanName, String triggerNam * @return String representation of the version number of Metastore server (eg: 3.1.0-SNAPSHOT) */ String getServerVersion() throws TException; + + /** + * Returns details about a scheduled query by name. + * + * @throws NoSuchObjectException if an object by the given name dosen't exists. + */ + ScheduledQuery getScheduledQuery(ScheduledQueryKey scheduleKey) throws TException; + + /** + * Carries out maintenance of scheduled queries (insert/update/drop). + */ + void scheduledQueryMaintenance(ScheduledQueryMaintenanceRequest request) throws MetaException, TException; + + /** + * Checks whenever a query is available for execution. + * + * @return optionally a scheduled query to be processed. + */ + ScheduledQueryPollResponse scheduledQueryPoll(ScheduledQueryPollRequest request) throws MetaException, TException; Review comment: I wonder whether we should be able to pull multiple queries at once? Maybe a possible extension... we can create a follow-up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324744) Time Spent: 2h 50m (was: 2h 40m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324710
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332279613
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -4215,17 +4222,27 @@ public Table alterTable(String catName, String dbname,
String name, Table newTab
*/
public static String verifyStatsChangeCtx(String fullTableName, Map oldP, Map newP,
long writeId, String
validWriteIds, boolean isColStatsChange) {
-if (validWriteIds != null && writeId > 0) return null; // We have txn
context.
+if (validWriteIds != null && writeId > 0)
+ {
Review comment:
Knitpick. Move to line above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324710)
Time Spent: 1h 20m (was: 1h 10m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324708
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332262473
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##
@@ -667,6 +667,19 @@ public static ConfVars getMetaConf(String name) {
RUNTIME_STATS_MAX_AGE("runtime.stats.max.age",
"hive.metastore.runtime.stats.max.age", 86400 * 3, TimeUnit.SECONDS,
"Stat entries which are older than this are removed."),
+SCHEDULED_QUERIES_CRON_SYNTAX("scheduled.queries.cron.syntax",
"hive.metastore.scheduled.queries.cron.syntax",
+"QUARTZ", new StringSetValidator("UNIX", "QUARTZ", "CRON4J", "SPRING"),
Review comment:
Is there a good reason to support all these different syntax? I am thinking
that maintaining / debugging them may become hard. I think we should stick to
the most expressive one and support that one, unless there is some value on
supporting all of them...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324708)
Time Spent: 1h 20m (was: 1h 10m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324743 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332256997 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/MetastoreBasedScheduledQueryService.java ## @@ -0,0 +1,64 @@ +package org.apache.hadoop.hive.ql.schq; Review comment: Missing license. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324743) Time Spent: 2h 50m (was: 2h 40m) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324753 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r331846838 ## File path: metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql ## @@ -1552,3 +1611,41 @@ WHERE AND (P.`PRINCIPAL_NAME`=current_user() AND P.`PRINCIPAL_TYPE`='USER' OR ((array_contains(current_groups(), P.`PRINCIPAL_NAME`) OR P.`PRINCIPAL_NAME` = 'public') AND P.`PRINCIPAL_TYPE`='GROUP')) AND P.`TBL_PRIV`='SELECT' AND P.`AUTHORIZER`=current_authorizer()); + + +drop view SCHEDULED_QUERIES ; +create or replace view SCHEDULED_QUERIES as Review comment: We should not add these views to the information schema database which should be as close to the standard as possible. I think you did this because the `from_unixtime` function cannot be applied when an external table is created? However, you can declare the time columns in the external tables with `timestamp with local time zone` type. Additionally, that will give the user the readable datetime instant. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324753) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324752 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r331673256 ## File path: itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java.orig ## @@ -0,0 +1,1053 @@ +/* Review comment: This file can be removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324752) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324738 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332259393 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/IScheduledQueryMaintenanceService.java ## @@ -0,0 +1,19 @@ +package org.apache.hadoop.hive.ql.schq; + +import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse; +import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo; + +/** + * Interface to cover Scheduled Query source service + * + * Note that the main reason to have this layer is to make testing easier. Review comment: Since this is only used for testing purposes, I am not sure we need it... Why don't we just create the class that is needed for testing as a service that overrides `MetastoreBasedScheduledQueryService`? Otherwise, having an interface is not bad, but maybe you can delete this line in the comment... we may have a new implementation of the service in the future :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324738) Time Spent: 2h 40m (was: 2.5h) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 40m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324719 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332276521 ## File path: ql/src/test/queries/clientpositive/sysdb_schq.q ## @@ -0,0 +1,38 @@ +--! qt:scheduledqueryservice +--! qt:dataset:src + +source ../../metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql; + +use sys; + +create scheduled query asd cron '* * * * * ? *' defined as select 1; + +!sleep 5; Review comment: This option was not existing before, right? This is very useful, we should document it! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324719) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324742
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332154141
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
##
@@ -2066,6 +2074,71 @@ dropMaterializedViewStatement
: KW_DROP KW_MATERIALIZED KW_VIEW ifExists? viewName ->
^(TOK_DROP_MATERIALIZED_VIEW viewName ifExists?)
;
+createScheduledQueryStatement
+@init { pushMsg("create scheduled query statement", state); }
+@after { popMsg(state); }
+: KW_CREATE KW_SCHEDULED KW_QUERY name=identifier
+scheduleSpec
+executedAsSpec?
Review comment:
I am not sure we are ready to support `executed as` in the syntax. One
problem that I see is that we do not check whether a certain user has
permissions to execute the query as another user. Probably some additional
checks and restrictions will be needed. We should just store the scheduled
query to be executed as the current user that created it, which we already do
below. We can work on the extensions in a follow-up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324742)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324707 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332256449 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ## @@ -166,6 +166,10 @@ private static BaseSemanticAnalyzer getInternal(QueryState queryState, ASTNode t case HiveParser.TOK_MERGE: return new MergeSemanticAnalyzer(queryState); + case HiveParser.TOK_ALTER_SCHEDULED_QUERY: + case HiveParser.TOK_CREATE_SCHEDULED_QUERY: + case HiveParser.TOK_DROP_SCHEDULED_QUERY: +return new ScheduledQueryAnalyzer(queryState); Review comment: knitpick. new line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324707) Time Spent: 1h 10m (was: 1h) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 1h 10m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324726 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332256819 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/IScheduledQueryMaintenanceService.java ## @@ -0,0 +1,19 @@ +package org.apache.hadoop.hive.ql.schq; Review comment: Missing license. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324726) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 10m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324702
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r330129801
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4771,6 +4772,19 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
HIVE_QUERY_PLANMAPPER_LINK_RELNODES("hive.query.planmapper.link.relnodes",
true,
"Wether to link Calcite nodes to runtime statistics."),
+
HIVE_SCHEDULED_QUERIES_EXECUTOR_ENABLED("hive.scheduled.queries.executor.enabled",
true,
+"Controls wether HS2 will run scheduled query executor."),
Review comment:
typo: wether -> whether
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324702)
Time Spent: 0.5h (was: 20m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324739 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 07/Oct/19 23:46 Start Date: 07/Oct/19 23:46 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794#discussion_r332261097 ## File path: ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionContext.java ## @@ -0,0 +1,36 @@ +package org.apache.hadoop.hive.ql.schq; Review comment: Missing license. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 324739) Time Spent: 2h 40m (was: 2.5h) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 2h 40m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324701
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331670594
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/schq/TestScheduledQueryIntegration.java
##
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.schq;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.Optional;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.ObjectStore;
+import org.apache.hadoop.hive.metastore.PersistenceManagerProvider;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import org.apache.hadoop.hive.metastore.model.MScheduledQuery;
+import org.apache.hadoop.hive.ql.DriverFactory;
+import org.apache.hadoop.hive.ql.IDriver;
+import org.apache.hadoop.hive.ql.parse.ParseException;
+import org.apache.hadoop.hive.ql.processors.CommandProcessorResponse;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryExecutionService;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.testutils.HiveTestEnvSetup;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+
+public class TestScheduledQueryIntegration {
Review comment:
Can we add a short comment on the purpose of this test?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324701)
Time Spent: 20m (was: 10m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324714
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331843882
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestOptionDispatcher.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.qoption;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.hive.ql.QTestUtil;
+
+public class QTestOptionDispatcher {
Review comment:
Can we add a comment about the different options that are currently
supported?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324714)
Time Spent: 1h 40m (was: 1.5h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324750
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332263885
##
File path:
standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
##
@@ -1754,6 +1754,57 @@ struct CreateTableRequest {
10: optional string processorIdentifier
}
+struct ScheduledQueryPollRequest {
+ 1: required string clusterNamespace
+}
+
+struct ScheduledQueryPollResponse {
+ 1: optional ScheduledQueryKey scheduleKey,
+ 2: optional i64 executionId,
+ 3: optional string query,
+ 4: optional string user,
+}
+
+struct ScheduledQueryKey {
+ 1: required string scheduleName,
+ 2: required string clusterNamespace,
+}
+
+struct ScheduledQuery {
+ 1: required ScheduledQueryKey scheduleKey,
+ 2: optional bool enabled,
+ 4: optional string schedule,
+ 5: optional string user,
+ 6: optional string query,
+ 7: optional i32 nextExecution,
+}
+
+enum ScheduledQueryMaintenanceRequestType {
+INSERT = 1,
Review comment:
We can change the enum keys here (left similar comment above).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324750)
Time Spent: 2h 50m (was: 2h 40m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324727
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332279696
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -4215,17 +4222,27 @@ public Table alterTable(String catName, String dbname,
String name, Table newTab
*/
public static String verifyStatsChangeCtx(String fullTableName, Map oldP, Map newP,
long writeId, String
validWriteIds, boolean isColStatsChange) {
-if (validWriteIds != null && writeId > 0) return null; // We have txn
context.
+if (validWriteIds != null && writeId > 0)
+ {
+ return null; // We have txn context.
+}
String oldVal = oldP == null ? null :
oldP.get(StatsSetupConst.COLUMN_STATS_ACCURATE);
String newVal = newP == null ? null :
newP.get(StatsSetupConst.COLUMN_STATS_ACCURATE);
// We don't need txn context is that stats state is not being changed.
-if (StringUtils.isEmpty(oldVal) && StringUtils.isEmpty(newVal)) return
null;
+if (StringUtils.isEmpty(oldVal) && StringUtils.isEmpty(newVal)) {
+ return null;
+}
if (StringUtils.equalsIgnoreCase(oldVal, newVal)) {
- if (!isColStatsChange) return null; // No change in col stats or
parameters => assume no change.
+ if (!isColStatsChange)
+ {
Review comment:
Knitpick. Move to line above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324727)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324703
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331673021
##
File path: itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
##
@@ -266,6 +285,25 @@ public void clearKeysCreatedInTests() {
}
}
+ public void clearScheduledQueries() {
+if (System.getenv(QTEST_LEAVE_FILES) != null) {
+ return;
+}
+Set scheduledQueries =
ScheduledQueryCreationRegistryHook.getSchedules();
+for (String name : scheduledQueries) {
+ ScheduledQueryMaintenanceRequest request = new
ScheduledQueryMaintenanceRequest();
+ request.setType(ScheduledQueryMaintenanceRequestType.DELETE);
+ ScheduledQuery schq = new ScheduledQuery();
+ schq.setScheduleKey(new ScheduledQueryKey(name,
conf.getVar(ConfVars.HIVE_SCHEDULED_QUERIES_NAMESPACE)));
+ request.setScheduledQuery(schq);
+ try {
+db.getMSC().scheduledQueryMaintenance(request);
+ } catch (Exception e) {
+LOG.error("Cant remove scheduled query: " + name + " " +
e.getMessage());
Review comment:
Typo: `Cant`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324703)
Time Spent: 40m (was: 0.5h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324722
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332264490
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionContext.java
##
@@ -0,0 +1,36 @@
+package org.apache.hadoop.hive.ql.schq;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+
+public class ScheduledQueryExecutionContext {
Review comment:
Comment on the class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324722)
Time Spent: 2h (was: 1h 50m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324717
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332256744
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
##
@@ -196,7 +196,11 @@
DROP_POOL("DROP POOL", HiveParser.TOK_DROP_POOL, null, null, false, false),
CREATE_MAPPING("CREATE MAPPING", HiveParser.TOK_CREATE_MAPPING, null, null,
false, false),
ALTER_MAPPING("ALTER MAPPING", HiveParser.TOK_ALTER_MAPPING, null, null,
false, false),
- DROP_MAPPING("DROP MAPPING", HiveParser.TOK_DROP_MAPPING, null, null, false,
false);
+ DROP_MAPPING("DROP MAPPING", HiveParser.TOK_DROP_MAPPING, null, null, false,
false),
+ CREATE_SCHEDULED_QUERY("CREATE SCHEDULED QUERY",
HiveParser.TOK_CREATE_SCHEDULED_QUERY, null, null),
+ ALTER_SCHEDULED_QUERY("ALTER SCHEDULED QUERY",
HiveParser.TOK_ALTER_SCHEDULED_QUERY, null, null),
+ DROP_SCHEDULED_QUERY("DROP SCHEDULED QUERY",
HiveParser.TOK_DROP_SCHEDULED_QUERY, null, null),
Review comment:
knitpick. you can drop the comma.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324717)
Time Spent: 1h 40m (was: 1.5h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324737
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332244066
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ScheduledQueryAnalyzer.java
##
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import static org.apache.hadoop.hive.ql.parse.HiveParser.TOK_IFEXISTS;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.Stack;
+
+import org.antlr.runtime.tree.Tree;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery;
+import org.apache.hadoop.hive.metastore.api.ScheduledQuery._Fields;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryKey;
+import
org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequestType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
+import org.apache.hadoop.hive.ql.exec.FunctionUtils;
+import org.apache.hadoop.hive.ql.exec.TaskFactory;
+import org.apache.hadoop.hive.ql.hooks.WriteEntity;
+import org.apache.hadoop.hive.ql.lib.Dispatcher;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.PreOrderWalker;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.HiveOperation;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ScheduledQueryAnalyzer extends BaseSemanticAnalyzer {
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryAnalyzer.class);
+
+ public ScheduledQueryAnalyzer(QueryState queryState) throws
SemanticException {
+super(queryState);
+ }
+
+ @Override
+ public void analyzeInternal(ASTNode ast) throws SemanticException {
+ScheduledQueryMaintWork work;
+ScheduledQueryMaintenanceRequestType type =
translateAstType(ast.getToken().getType());
+ScheduledQuery schq = interpretAstNode(ast);
+fillScheduledQuery(type, schq);
+LOG.info("scheduled query operation: " + type + " " + schq);
+try {
+ schq.validate();
+} catch (TException e) {
+ throw new SemanticException("ScheduledQuery is invalid", e);
+}
+work = new ScheduledQueryMaintWork(type, schq);
+rootTasks.add(TaskFactory.get(work));
+ }
+
+ private void fillScheduledQuery(ScheduledQueryMaintenanceRequestType type,
ScheduledQuery schq)
+ throws SemanticException {
+if (type == ScheduledQueryMaintenanceRequestType.INSERT) {
+ populateUnfilled(schq, buildEmptySchq());
Review comment:
The call to this method seems to cause unnecessary overhead. We could just
set the needed fields to the corresponding value rather than iterating through
the fields?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324715
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332260987
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/schq/MetastoreBasedScheduledQueryService.java
##
@@ -0,0 +1,64 @@
+package org.apache.hadoop.hive.ql.schq;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollRequest;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class MetastoreBasedScheduledQueryService implements
IScheduledQueryMaintenanceService {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MetastoreBasedScheduledQueryService.class);
+ private HiveConf conf;
+
+ public MetastoreBasedScheduledQueryService(HiveConf conf) {
+this.conf = conf;
+ }
+
+ @Override
+ public ScheduledQueryPollResponse scheduledQueryPoll() {
+try {
+ ScheduledQueryPollRequest request = new ScheduledQueryPollRequest();
+ request.setClusterNamespace(getClusterNamespace());
+ ScheduledQueryPollResponse resp = getMSC().scheduledQueryPoll(request);
+ return resp;
+} catch (Exception e) {
+ logException("Exception while polling scheduled queries", e);
+ return null;
+}
+ }
+
+ @Override
+ public void scheduledQueryProgress(ScheduledQueryProgressInfo info) {
+try {
+ getMSC().scheduledQueryProgress(info);
+} catch (Exception e) {
+ logException("Exception while updating scheduled execution status of: "
+ info.getScheduledExecutionId(), e);
+}
+ }
+
+ private IMetaStoreClient getMSC() throws MetaException, HiveException {
+return Hive.get(conf).getMSC();
+ }
+
+ static void logException(String msg, Exception e) {
+if (LOG.isDebugEnabled()) {
Review comment:
This does not seem intuitive... Let's log it as error? Is it expected in any
case to get an exception over here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324715)
Time Spent: 1h 40m (was: 1.5h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324706
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r331850162
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintTask.java
##
@@ -0,0 +1,46 @@
+package org.apache.hadoop.hive.ql.exec.schq;
+
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryMaintenanceRequest;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+import org.apache.hadoop.hive.ql.schq.ScheduledQueryMaintWork;
+import org.apache.thrift.TException;
+
+public class ScheduledQueryMaintTask extends Task {
Review comment:
Please add javadoc description to class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324706)
Time Spent: 1h 10m (was: 1h)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch,
> HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch,
> HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch,
> HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch,
> HIVE-21884.20.patch, Scheduled queries2.pdf
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=324730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324730
]
ASF GitHub Bot logged work on HIVE-21884:
-
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:46
Start Date: 07/Oct/19 23:46
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #794: HIVE-21884
URL: https://github.com/apache/hive/pull/794#discussion_r332275454
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionService.java
##
@@ -0,0 +1,159 @@
+package org.apache.hadoop.hive.ql.schq;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.QueryState;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryPollResponse;
+import org.apache.hadoop.hive.metastore.api.ScheduledQueryProgressInfo;
+import org.apache.hadoop.hive.ql.DriverFactory;
+import org.apache.hadoop.hive.ql.IDriver;
+import org.apache.hadoop.hive.ql.processors.CommandProcessorResponse;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class ScheduledQueryExecutionService implements Closeable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ScheduledQueryExecutionService.class);
+
+ private ScheduledQueryExecutionContext context;
+ private ScheduledQueryExecutor worker;
+
+ public static ScheduledQueryExecutionService
startScheduledQueryExecutorService(HiveConf conf) {
+MetastoreBasedScheduledQueryService qService = new
MetastoreBasedScheduledQueryService(conf);
+ExecutorService executor =
+Executors.newCachedThreadPool(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Scheduled Query Thread
%d").build());
+ScheduledQueryExecutionContext ctx = new
ScheduledQueryExecutionContext(executor, conf, qService);
+return new ScheduledQueryExecutionService(ctx);
+ }
+
+ public ScheduledQueryExecutionService(ScheduledQueryExecutionContext ctx) {
+context = ctx;
+ctx.executor.submit(worker = new ScheduledQueryExecutor());
+ctx.executor.submit(new ProgressReporter());
+ }
+
+ class ScheduledQueryExecutor implements Runnable {
+
+private ScheduledQueryProgressInfo info;
+
+@Override
+public void run() {
+ while (true) {
+ScheduledQueryPollResponse q =
context.schedulerService.scheduledQueryPoll();
+if (q.isSetExecutionId()) {
+ try{
+processQuery(q);
+ } catch (Throwable t) {
+LOG.error("Unexpected exception during scheduled query
processing", t);
+ }
+} else {
+ try {
+Thread.sleep(context.getIdleSleepTime());
+ } catch (InterruptedException e) {
+Thread.currentThread().interrupt();
+LOG.info("interrupted");
+break;
+ }
+}
+ }
+}
+
+QueryState lastState;
+public synchronized void reportQueryProgress() {
+ if (info != null) {
+if(lastState == QueryState.FINISHED || lastState ==
QueryState.ERRORED) {
+ return;
+}
+LOG.info("Reporting query progress of {} as {} err:{}",
info.getScheduledExecutionId(), info.getState(),
+info.getErrorMessage());
+context.schedulerService.scheduledQueryProgress(info);
+lastState=info.getState();
+ }
+}
+
+private void processQuery(ScheduledQueryPollResponse q) {
Review comment:
What happens in the following two scenarios with current implementation?
1) A scheduled query is modified while it is being executed.
2) HS2 becomes irresponsive for a certain time, the grab on the lock
expires, another HS2 grabs the query execution, the first HS2 comes back to
life.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324730)
Time Spent: 2h 20m (was: 2h 10m)
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
[jira] [Commented] (HIVE-22239) Scale data size using column value ranges
[
https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946339#comment-16946339
]
Hive QA commented on HIVE-22239:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
31s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 26 new + 109 unchanged - 11 fixed = 135 total (was 120) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 15 new + 259 unchanged - 2
fixed = 274 total (was 261) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 170 unchanged - 0 fixed = 172 total (was 170) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} ql generated 0 new + 1550 unchanged - 1 fixed = 1550
total (was 1551) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Integral division result cast to double or float in
org.apache.hadoop.hive.metastore.columnstats.aggr.TimestampColumnStatsAggregator.aggregate(List,
List, boolean) At TimestampColumnStatsAggregator.java:double or float in
org.apache.hadoop.hive.metastore.columnstats.aggr.TimestampColumnStatsAggregator.aggregate(List,
List, boolean) At TimestampColumnStatsAggregator.java:[line 102] |
| |
org.apache.hadoop.hive.metastore.columnstats.cache.TimestampColumnStatsDataInspector
doesn't override
org.apache.hadoop.hive.metastore.api.TimestampColumnStatsData.equals(Object)
At TimestampColumnStatsDataInspector.java:At
[jira] [Commented] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946310#comment-16946310
]
Hive QA commented on HIVE-21924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982402/HIVE-21924.6.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17517 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18896/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18896/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18896/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982402 - PreCommit-HIVE-Build
> Split text files even if header/footer exists
> -
>
> Key: HIVE-21924
> URL: https://issues.apache.org/jira/browse/HIVE-21924
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21924.2.patch, HIVE-21924.3.patch,
> HIVE-21924.4.patch, HIVE-21924.5.patch, HIVE-21924.6.patch, HIVE-21924.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/967a1cc98beede8e6568ce750ebeb6e0d048b8ea/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494-L503
>
> {code}
> int headerCount = 0;
> int footerCount = 0;
> if (table != null) {
> headerCount = Utilities.getHeaderCount(table);
> footerCount = Utilities.getFooterCount(table, conf);
> if (headerCount != 0 || footerCount != 0) {
> // Input file has header or footer, cannot be splitted.
> HiveConf.setLongVar(conf, ConfVars.MAPREDMINSPLITSIZE,
> Long.MAX_VALUE);
> }
> }
> {code}
> this piece of code makes the CSV (or any text files with header/footer) files
> not splittable if header or footer is present.
> If only header is present, we can find the offset after first line break and
> use that to split. Similarly for footer, may be read few KB's of data at the
> end and find the last line break offset. Use that to determine the data range
> which can be used for splitting. Few reads during split generation are
> cheaper than not splitting the file at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946303#comment-16946303
]
Hive QA commented on HIVE-21924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18896/dev-support/hive-personality.sh
|
| git revision | master / e16877c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18896/yetus/patch-asflicense-problems.txt
|
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18896/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split text files even if header/footer exists
> -
>
> Key: HIVE-21924
> URL: https://issues.apache.org/jira/browse/HIVE-21924
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21924.2.patch, HIVE-21924.3.patch,
> HIVE-21924.4.patch, HIVE-21924.5.patch, HIVE-21924.6.patch, HIVE-21924.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/967a1cc98beede8e6568ce750ebeb6e0d048b8ea/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494-L503
>
> {code}
> int headerCount = 0;
> int footerCount = 0;
> if (table != null) {
> headerCount = Utilities.getHeaderCount(table);
> footerCount = Utilities.getFooterCount(table, conf);
> if (headerCount != 0 || footerCount != 0) {
> // Input file has header or footer, cannot be splitted.
> HiveConf.setLongVar(conf, ConfVars.MAPREDMINSPLITSIZE,
> Long.MAX_VALUE);
> }
> }
> {code}
> this piece of code makes the CSV
[jira] [Commented] (HIVE-22297) qtests: add regex based replacer
[
https://issues.apache.org/jira/browse/HIVE-22297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946259#comment-16946259
]
Hive QA commented on HIVE-22297:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982400/HIVE-22297.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18895/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18895/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18895/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12982400/HIVE-22297.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982400 - PreCommit-HIVE-Build
> qtests: add regex based replacer
>
>
> Key: HIVE-22297
> URL: https://issues.apache.org/jira/browse/HIVE-22297
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22297.01.patch
>
>
> right now we have some hard wired replacers for specific patterns
> idea would be to generalize it - and make it able to configure replacements
> from the qfile
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22297) qtests: add regex based replacer
[
https://issues.apache.org/jira/browse/HIVE-22297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946258#comment-16946258
]
Hive QA commented on HIVE-22297:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982400/HIVE-22297.01.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17515 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=120)
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
(batchId=225)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=234)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18894/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18894/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18894/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982400 - PreCommit-HIVE-Build
> qtests: add regex based replacer
>
>
> Key: HIVE-22297
> URL: https://issues.apache.org/jira/browse/HIVE-22297
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22297.01.patch
>
>
> right now we have some hard wired replacers for specific patterns
> idea would be to generalize it - and make it able to configure replacements
> from the qfile
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views
[
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946230#comment-16946230
]
Jesus Camacho Rodriguez commented on HIVE-21344:
[~vgarg], [~hashutosh], I have rebased this patch. Can you review it?
https://github.com/apache/hive/pull/749
Thanks
> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21344.01.patch, HIVE-21344.02.patch,
> HIVE-21344.03.patch, HIVE-21344.04.patch, HIVE-21344.patch,
> calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22235) CommandProcessorResponse should not be an exception
[ https://issues.apache.org/jira/browse/HIVE-22235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22235: -- Attachment: HIVE-22235.08.patch > CommandProcessorResponse should not be an exception > --- > > Key: HIVE-22235 > URL: https://issues.apache.org/jira/browse/HIVE-22235 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22235.01.patch, HIVE-22235.02.patch, > HIVE-22235.03.patch, HIVE-22235.04.patch, HIVE-22235.05.patch, > HIVE-22235.06.patch, HIVE-22235.07.patch, HIVE-22235.08.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The CommandProcessorResponse class extends Exception. This may be convenient, > but it is wrong, as a response is not an exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22136: --- Status: Open (was: Patch Available) > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22136: --- Status: Patch Available (was: Open) > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22136: --- Attachment: HIVE-22136.4.patch > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22297) qtests: add regex based replacer
[
https://issues.apache.org/jira/browse/HIVE-22297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946204#comment-16946204
]
Hive QA commented on HIVE-22297:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 1 new + 32 unchanged
- 0 fixed = 33 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18894/dev-support/hive-personality.sh
|
| git revision | master / e16877c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18894/yetus/diff-checkstyle-itests_util.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18894/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18894/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qtests: add regex based replacer
>
>
> Key: HIVE-22297
> URL: https://issues.apache.org/jira/browse/HIVE-22297
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22297.01.patch
>
>
> right now we have some hard wired replacers for specific patterns
> idea would be to generalize it - and make it able to configure replacements
> from the qfile
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22235) CommandProcessorResponse should not be an exception
[
https://issues.apache.org/jira/browse/HIVE-22235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946178#comment-16946178
]
Hive QA commented on HIVE-22235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982396/HIVE-22235.07.patch
{color:green}SUCCESS:{color} +1 due to 74 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17515 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testAuthZFailureLlapCachePurge
(batchId=287)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18893/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18893/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18893/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982396 - PreCommit-HIVE-Build
> CommandProcessorResponse should not be an exception
> ---
>
> Key: HIVE-22235
> URL: https://issues.apache.org/jira/browse/HIVE-22235
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22235.01.patch, HIVE-22235.02.patch,
> HIVE-22235.03.patch, HIVE-22235.04.patch, HIVE-22235.05.patch,
> HIVE-22235.06.patch, HIVE-22235.07.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The CommandProcessorResponse class extends Exception. This may be convenient,
> but it is wrong, as a response is not an exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22235) CommandProcessorResponse should not be an exception
[
https://issues.apache.org/jira/browse/HIVE-22235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946173#comment-16946173
]
Hive QA commented on HIVE-22235:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} cli in master has 9 extant Findbugs warnings. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
49s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
55s{color} | {color:red} ql: The patch generated 73 new + 1432 unchanged - 467
fixed = 1505 total (was 1899) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} cli: The patch generated 0 new + 39 unchanged - 2
fixed = 39 total (was 41) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hcatalog/core: The patch generated 0 new + 75
unchanged - 16 fixed = 75 total (was 91) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 183 unchanged - 11 fixed = 183 total (was 194) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch java-client
[jira] [Commented] (HIVE-22293) Additional test cases
[
https://issues.apache.org/jira/browse/HIVE-22293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946147#comment-16946147
]
Vineet Garg commented on HIVE-22293:
[~shortbread] Adding additional test cases would be great.
> Additional test cases
> -
>
> Key: HIVE-22293
> URL: https://issues.apache.org/jira/browse/HIVE-22293
> Project: Hive
> Issue Type: Improvement
>Reporter: Sumin Byeon
>Priority: Minor
>
> I've noticed some classes, such as {{DecimalColumnVector}},
> {{VoidColumnVector}}, {{TimestampColumnVector}}, do not have any test
> coverage (or insufficient coverage). Would it be okay if I write some test
> code and submit patches or pull requests?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946119#comment-16946119
]
Hive QA commented on HIVE-22284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982393/HIVE-22284.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17517 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18892/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18892/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18892/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982393 - PreCommit-HIVE-Build
> Improve LLAP CacheContentsTracker to collect and display correct statistics
> ---
>
> Key: HIVE-22284
> URL: https://issues.apache.org/jira/browse/HIVE-22284
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch,
> HIVE-22284.2.patch, HIVE-22284.3.patch
>
>
> When keeping track of which buffers correspond to what Hive objects,
> CacheContentsTracker relies on cache tags.
> Currently a tag is a simple String that ideally holds DB and table name, and
> a partition spec concatenated by . and / . The information here is derived
> from the Path of the file that is getting cached. Needless to say sometimes
> this produces a wrong tag especially for external tables.
> Also there's a bug when calculating aggregated stats for a 'parent' tag
> (corresponding to the table of the partition) because the overall maxCount
> and maxSize do not add up to the sum of those in the partitions. This happens
> when buffers get removed from the cache.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946080#comment-16946080
]
Hive QA commented on HIVE-22284:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 10 new + 4 unchanged
- 0 fixed = 14 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 4 new + 165 unchanged - 2
fixed = 169 total (was 167) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 1 new + 252 unchanged
- 13 fixed = 253 total (was 265) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} storage-api generated 1 new + 28 unchanged - 0 fixed =
29 total (was 28) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18892/dev-support/hive-personality.sh
|
| git revision | master / e16877c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18892/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18892/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18892/yetus/diff-checkstyle-llap-server.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18892/yetus/diff-javadoc-javadoc-storage-api.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18892/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api llap-common ql llap-server U: . |
| Console output |
[jira] [Updated] (HIVE-22239) Scale data size using column value ranges
[ https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22239: --- Attachment: HIVE-22239.01.patch > Scale data size using column value ranges > - > > Key: HIVE-22239 > URL: https://issues.apache.org/jira/browse/HIVE-22239 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22239.01.patch, HIVE-22239.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, min/max values for columns are only used to determine whether a > certain range filter falls out of range and thus filters all rows or none at > all. If it does not, we just use a heuristic that the condition will filter > 1/3 of the input rows. Instead of using that heuristic, we can use another > one that assumes that data will be uniformly distributed across that range, > and calculate the selectivity for the condition accordingly. > This patch also includes the propagation of min/max column values from > statistics to the optimizer for timestamp type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22252: Attachment: HIVE-22252.4.patch > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch, > HIVE-22252.3.patch, HIVE-22252.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22249) Support Parquet through HCatalog
[
https://issues.apache.org/jira/browse/HIVE-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946050#comment-16946050
]
Hive QA commented on HIVE-22249:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982389/HIVE-22249.3.branch-2.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testDynamicPartitioningMultiPartColsInDataNoSpec
(batchId=179)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18891/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18891/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18891/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982389 - PreCommit-HIVE-Build
> Support Parquet through HCatalog
>
>
> Key: HIVE-22249
> URL: https://issues.apache.org/jira/browse/HIVE-22249
> Project: Hive
> Issue Type: New Feature
>Reporter: Jay Green-Stevens
>Assignee: Jay Green-Stevens
>Priority: Major
> Fix For: 2.3.7
>
> Attachments: HIVE-22249.2.branch-2.3.patch,
> HIVE-22249.3.branch-2.patch, HIVE-22249.branch-2.3.patch
>
>
> HIVE-8838 added Parquet support to HCatalog for Hive 3.0.0. We would like to
> backport this functionality to Hive 2.x (primarily 2.3.x) for users who are
> currently unable to migrate to Hive 3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22250) Describe function does not provide description for rank functions
[
https://issues.apache.org/jira/browse/HIVE-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22250:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~kkasa]!
> Describe function does not provide description for rank functions
> -
>
> Key: HIVE-22250
> URL: https://issues.apache.org/jira/browse/HIVE-22250
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22250.1.patch, HIVE-22250.1.patch,
> HIVE-22250.1.patch, HIVE-22250.2.patch, HIVE-22250.3.patch,
> HIVE-22250.3.patch, HIVE-22250.4.patch, HIVE-22250.5.patch
>
>
> {code}
> @WindowFunctionDescription(
> description = @Description(
> name = "dense_rank",
> value = "_FUNC_(x) The difference between RANK and DENSE_RANK is that
> DENSE_RANK leaves no " +
> "gaps in ranking sequence when there are ties. That is, if you
> were " +
> "ranking a competition using DENSE_RANK and had three people tie
> for " +
> "second place, you would say that all three were in second place
> and " +
> "that the next person came in third."
> ),
> supportsWindow = false,
> pivotResult = true,
> rankingFunction = true,
> impliesOrder = true
> )
> {code}
> {code}
> DESC FUNCTION dense_rank;
> {code}
> {code}
> PREHOOK: query: DESC FUNCTION dense_rank
> PREHOOK: type: DESCFUNCTION
> POSTHOOK: query: DESC FUNCTION dense_rank
> POSTHOOK: type: DESCFUNCTION
> There is no documentation for function 'dense_rank'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22281:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~kkasa]!
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch, HIVE-22281.2.patch,
> HIVE-22281.3.patch, HIVE-22281.3.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22300) Deduplicate the authentication and LDAP code in HMS and HS2
[ https://issues.apache.org/jira/browse/HIVE-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat reassigned HIVE-22300: - > Deduplicate the authentication and LDAP code in HMS and HS2 > --- > > Key: HIVE-22300 > URL: https://issues.apache.org/jira/browse/HIVE-22300 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > > HIVE-22267 has duplicated code from hive-service/auth directory under > standalone-metastore directory. Deduplicate this code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[ https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22248: --- Fix Version/s: 4.0.0 > Min value for column in stats is not set correctly for some data types > -- > > Key: HIVE-22248 > URL: https://issues.apache.org/jira/browse/HIVE-22248 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch, > HIVE-22248.04.patch > > Time Spent: 50m > Remaining Estimate: 0h > > I am not sure whether the problem is printing the value or in the value > stored in the metastore itself, but for some types (e.g. tinyint, smallint, > int, bigint, double or float), the min value does not seem to be set > correctly (set to 0). > https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types
[
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946002#comment-16946002
]
Hive QA commented on HIVE-21407:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982386/HIVE-21407.4.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 17320 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=234)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=234)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18890/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18890/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18890/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982386 - PreCommit-HIVE-Build
> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch,
> HIVE-21407.4.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate
> is not pushed to parquet, so the filtering only happens within Hive. If the
> parameter is true, the filter is pushed to parquet, but for a char type, the
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
> @Override
> public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
> }
> {noformat}
> So if we have a char(10) column which contains the value "apple" and the
> where condition looks like 'where c='apple'', the value pushed to Paquet will
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello',
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c | ppd.v | ppd.i |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v | ppd.i |
> +-+++
> | apple | bee| 1 |
> | apple | tree | 2 |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded
> upstream. Before the HIVE-21327 fix, setting the parameter
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q
> test hid this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
