[jira] [Updated] (HIVE-25647) hadoop memo
[
https://issues.apache.org/jira/browse/HIVE-25647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
St Li updated HIVE-25647:
-
Description:
master,slave1,slave2master,slave1,slave2//opt represent wechat hadoop bigdata
dev//2019 :bigdata competitionhadoop 50070hbase 16010storm 8080
#hostnamehostnamectl set-hostname master && bash hostname master && bash
hostname slave1/slave2 && bash vim /etc/hostname master/slave1/slave2vim
/etc/hosts ip master ip slave1 ipslave2
#yumcd /etc/yum.repos.d && rm -rf *wget
http://172.16.47.240/bigdata/repofile/bigdata.repoyum clean all
#firewallsystemctl stop firewalldsystemctl status firewalld
#timezonetzselect 5-9-1-1echo "TZ='Asia/Shanghai'; export TZ" >> /etc/profile
&& source /etc/profile
#ntpyum install -y ntpvim /etc/ntp.conf//#server 0~3.centos.pool.ntp.org
iburstserver 127.127.1.0fudge 127.127.1.0 stratum 10/bin/systemctl restart
ntpd.servicentpdate master (slave1,slave2)
#crontabservice crond status/sbin/service crond startcrontab -e*/30 8-17 * * *
/usr/sbin/ntpdate mastercrontab –l
#ssh passwordssh-keygen -t dsa -P '' -f ~/.ssh/id_dsacat /root/.ssh/id_dsa.pub
>> /root/.ssh/authorized_keysscp ~/.ssh/authorized_keys root@slave1:~/.ssh/scp
~/.ssh/authorized_keys root@slave2:~/.ssh/
ssh-copy-id masterssh-copy-id slave1ssh-copy-id slave2
#install jdkmkdir -p /usr/javatar -zxvf jdk-8u171-linux-x64.tar.gz -C /usr/java/
vim /etc/profileexport JAVA_HOME=/usr/java/jdk1.8.0_171export
CLASSPATH=$JAVA_HOME/lib/export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile && java -version
scp -r /usr/java root@slave1:/usr/scp -r /usr/java root@slave2:/usr/
#install hadoopmkdir -p /usr/hadoop && cd /usr/hadooptar -zxvf
/usr/hadoop/hadoop-2.7.3.tar.gz -C /usr/hadoop/rm -rf
/usr/hadoop/hadoop-2.7.3.tar.gzvim /etc/profileexport
HADOOP_HOME=/usr/hadoop/hadoop-2.7.3export
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbinhadoop //test
hadoop-env.sh/mapred-env.sh/yarn-en.shexport JAVA_HOME=/usr/java/jdk1.8.0_171
##vim core-site.xmlfs.default.name \{hdfs://master:9000}hadoop.tmp.dir
\{/usr/hadoop/hadoop-2.7.3/hdfs/tmp}io.file.buffer.size
\{131072}fs.checkpoint.period \{60}fs.checkpoint.size \{67108864}
##hdfs-site.xmldfs.replication \{2}dfs.namenode.name.dir
\{file:/usr/hadoop/hadoop-2.7.3/hdfs/name}dfs.datanode.data.dir
\{file:/usr/hadoop/hadoop-2.7.3/hdfs/data}
##vim yarn-env.shyarn.resourcemanager.address
\{master:18040}yarn.resourcemanager.scheduler.address
\{master:18030}yarn.resourcemanager.webapp.address
\{master:18088}yarn.resourcemanager.resource-tracker.address
\{18025}yarn.resourcemanager.admin.address
\{master:18141}yarn.nodemanager.aux-services
\{mapreduce_shuffle}yarn.nodemanager.auxservices.mapreduce.shuffle.class
\{org.apache.hadoop.mapred.ShuffleHandler}
#vim mapred-site.xmlmapreduce.framework.name \{yarn}
#slaves fileecho master > master && echo slave1 > slaves && echo slave2 >>
slaves
#hadoop formathadoop namenode -format (master) //has been successfully#start
hadoopstart-all.shmaster
:NameNode,SecondaryNameNode,ResourceManagerslave1~2:DataNode,NodeManager
start-dfs.shstart-yarn.shhadoop-daemon.sh start namenodehadoop-daemon.sh start
datanodehadoop-daemon.sh start secondarynamenodehadoop-daemon.sh start
resourcemanagerhadoop-daemon.sh start nodemanager
test hdfs& mapreducehadoop fs -mkdir /inputhadoop fs -put
$HADOOP_HOME/README.txt /input
http://master:50070hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.4.jar
#install hiveyum -y install mysql-community-server slave2:mysqlserver
slave1:hiveserver master:hiveclientsystemctl daemon-reloadsystemctl start
mysqldcat /var/log/mysqld.log grep "temporary password"mysql -uroot -pset
global validate_password_policy=0;set global validate_password_length=4;alter
user 'root'@'localhost' identified by '123456';mysql -uroot -p123456create user
'root'@'%' identified by '123456';grant all privileges on *.* to 'root'@'%'
with grant option;flush privileges;
mkdir -p /usr/hive tar -zxvf /usr/hive/apache-hive-2.1.1-bin.tar.gz -C
/usr/hive/
vim /etc/profile //for hiveexport
HIVE_HOME=/usr/hive/apache-hive-2.1.1-binexport PATH=$PATH:$HIVE_HOME/binsource
/etc/profile
vim hive-env.shcd $HIVE_HOME/conf && vim hive-env.shexport
HADOOP_HOME=/usr/hadoop/hadoop-2.7.3export
HIVE_CONF_DIR=/usr/hive/apache-hive-2.1.1-bin/confexport
HIVE_AUX_JARS_PATH=/usr/hive/apache-hive-2.1.1-bin/lib
cp $HIVE_HOME/lib/jline-2.12.jar $HADOOP_HOME/share/hadoop/yarn/lib/
##slave1 hive-servercd $HIVE_HOME/lib && wget or cp
mysql-connector-java-5.1.47-bin.jar
hive-site.xml (hive-server)hive.metastore.warehouse.dir
\{/user/hive_remote/warehouse}javax.jdo.option.ConnectionDriverName
\{com.mysql.jdbc.Driver}javax.jdo.option.ConnectionURL
\{jdbc:mysql://slave2:3306/hive?createDatabaseIfNotExist=true&useSSL=false}javax.jdo.option.ConnectionUserName
\{root}javax.jdo.option.ConnectionPassword \{123456}
hive-site.xml (hive client)hive.metasto
[jira] [Work logged] (HIVE-25616) Backport HIVE-24741 to Hive 2, 3
[
https://issues.apache.org/jira/browse/HIVE-25616?focusedWorklogId=671201&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671201
]
ASF GitHub Bot logged work on HIVE-25616:
-
Author: ASF GitHub Bot
Created on: 28/Oct/21 04:48
Start Date: 28/Oct/21 04:48
Worklog Time Spent: 10m
Work Description: nssalian commented on pull request #2733:
URL: https://github.com/apache/hive/pull/2733#issuecomment-953499315
@vihangk1 can we merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 671201)
Time Spent: 2h 20m (was: 2h 10m)
> Backport HIVE-24741 to Hive 2, 3
> -
>
> Key: HIVE-25616
> URL: https://issues.apache.org/jira/browse/HIVE-25616
> Project: Hive
> Issue Type: Improvement
>Reporter: Neelesh Srinivas Salian
>Assignee: Neelesh Srinivas Salian
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> HIVE-24741 adds a major improvement to the `{{get_partitions_ps_with_auth}}`
> API that is used by Spark to retrieve all partitions of the table.
> This has caused problems in Spark 3 - running on a Hive 2.3.x metastore.
> Patched this in my org and it helped get past large partitioned tables having
> problems reading metadata through Spark.
> [~vihangk1], thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25636) Bump Xerce2 to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-25636?focusedWorklogId=671160&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671160 ] ASF GitHub Bot logged work on HIVE-25636: - Author: ASF GitHub Bot Created on: 28/Oct/21 01:58 Start Date: 28/Oct/21 01:58 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #2740: URL: https://github.com/apache/hive/pull/2740#issuecomment-953438106 @kgyrtkirk Thanks for your review. Is there anything I should do to merge it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 671160) Time Spent: 20m (was: 10m) > Bump Xerce2 to 2.12.1 > - > > Key: HIVE-25636 > URL: https://issues.apache.org/jira/browse/HIVE-25636 > Project: Hive > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Xerce 2.9.1 has high-level vulnerabilities. > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4002] > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-0881] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-2462) make INNER a non-reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-2462?focusedWorklogId=671135&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671135 ] ASF GitHub Bot logged work on HIVE-2462: Author: ASF GitHub Bot Created on: 28/Oct/21 00:10 Start Date: 28/Oct/21 00:10 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2462: URL: https://github.com/apache/hive/pull/2462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 671135) Time Spent: 20m (was: 10m) > make INNER a non-reserved keyword > - > > Key: HIVE-2462 > URL: https://issues.apache.org/jira/browse/HIVE-2462 > Project: Hive > Issue Type: Improvement >Reporter: John Sichi >Assignee: John Sichi >Priority: Major > Labels: pull-request-available > Fix For: 0.8.0 > > Attachments: HIVE-2462.1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > HIVE-2191 introduced the INNER keyword as reserved, which breaks backwards > compatibility for queries which were using it as an identifier. This patch > addresses that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25528) Avoid recalculating types after CBO on second AST pass
[ https://issues.apache.org/jira/browse/HIVE-25528?focusedWorklogId=670928&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670928 ] ASF GitHub Bot logged work on HIVE-25528: - Author: ASF GitHub Bot Created on: 27/Oct/21 17:58 Start Date: 27/Oct/21 17:58 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #2722: URL: https://github.com/apache/hive/pull/2722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 670928) Time Spent: 3h 10m (was: 3h) > Avoid recalculating types after CBO on second AST pass > -- > > Key: HIVE-25528 > URL: https://issues.apache.org/jira/browse/HIVE-25528 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > It should be possible to avoid recalculating and reevaluating types on the > second pass after going through CBO. CBO is making the effort to change the > types so to reassess them is a waste of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23556) Support hive.metastore.limit.partition.request for get_partitions_ps
[ https://issues.apache.org/jira/browse/HIVE-23556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434911#comment-17434911 ] Naveen Gangam commented on HIVE-23556: -- [~touchida] Is there a PR for this? Can you please rebase the PR or create a new one please? Thank you > Support hive.metastore.limit.partition.request for get_partitions_ps > > > Key: HIVE-23556 > URL: https://issues.apache.org/jira/browse/HIVE-23556 > Project: Hive > Issue Type: Improvement >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-23556.2.patch, HIVE-23556.3.patch, > HIVE-23556.4.patch, HIVE-23556.patch > > > HIVE-13884 added the configuration hive.metastore.limit.partition.request to > limit the number of partitions that can be requested. > Currently, it takes in effect for the following MetaStore APIs > * get_partitions, > * get_partitions_with_auth, > * get_partitions_by_filter, > * get_partitions_spec_by_filter, > * get_partitions_by_expr, > but not for > * get_partitions_ps, > * get_partitions_ps_with_auth. > This issue proposes to apply the configuration also to get_partitions_ps and > get_partitions_ps_with_auth. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-25637) Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
[
https://issues.apache.org/jira/browse/HIVE-25637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434895#comment-17434895
]
László Bodor edited comment on HIVE-25637 at 10/27/21, 3:16 PM:
Finally, it turned out we need 2 changes:
1. Consider all TableDesc objects for kafka credentials which are present in
FileSinkOperators in MapWork. This is because HIVE-23408 has taken care of
TableDescs of the source table, but while inserting, the source table is a
dummy table, explain is like:
{code}
| Stage: Stage-2 |
| Tez|
| DagId: hive_20211026130429_4c237196-f3ef-4edb-8cc5-584d63f8b03d:4 |
| DagName: hive_20211026130429_4c237196-f3ef-4edb-8cc5-584d63f8b03d:4 |
| Vertices:|
| Map 1 |
| Map Operator Tree: |
| TableScan |
| alias: _dummy_table |
| Row Limit Per Split: 1 |
| Statistics: Num rows: 1 Data size: 10 Basic stats: COMPLETE
Column stats: COMPLETE |
| Select Operator |
| expressions: array(const
struct(null,'comment',0,1,2,3,null,null,null,null)) (type:
array>)
|
| outputColumnNames: _col0 |
| Statistics: Num rows: 1 Data size: 48 Basic stats:
COMPLETE Column stats: COMPLETE |
| UDTF Operator |
| Statistics: Num rows: 1 Data size: 48 Basic stats:
COMPLETE Column stats: COMPLETE |
| function name: inline|
| Select Operator |
| expressions: null (type: timestamp), col2 (type:
string), UDFToBoolean(col3) (type: boolean), col4 (type: int), UDFToLong(col5)
(type: bigint), UDFToDouble(col6) (type: double), null (type: binary), null
(type: int), null (type: bigint), null (type: bigint) |
| outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7, _col8, _col9 |
| Statistics: Num rows: 1 Data size: 120 Basic stats:
COMPLETE Column stats: COMPLETE |
| File Output Operator |
| compressed: false|
| Statistics: Num rows: 1 Data size: 120 Basic stats:
COMPLETE Column stats: COMPLETE |
| table: |
| input format:
org.apache.hadoop.hive.kafka.KafkaInputFormat |
| output format:
org.apache.hadoop.hive.kafka.KafkaOutputFormat |
| serde: org.apache.hadoop.hive.kafka.KafkaSerDe |
| name: default.kafka_table |
||
{code}
2. merge credentials in TezProcessor to make them available for Hive operators,
because there were no tokens present in below codepath -> this caused a
fallback to GSSAPI (kerberos auth) instead of token based auth in KafkaUtils
{code}
at
org.apache.hadoop.hive.kafka.KafkaUtils.addKerberosJaasConf(KafkaUtils.java:362)
at
org.apache.hadoop.hive.kafka.KafkaUtils.producerProperties(KafkaUtils.java:230)
at
org.apache.hadoop.hive.kafka.KafkaOutputFormat.getHiveRecordWriter(KafkaOutputFormat.java:56)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:294)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:279)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:872)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:823)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1004)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at
org.apach
[jira] [Commented] (HIVE-25637) Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
[
https://issues.apache.org/jira/browse/HIVE-25637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434895#comment-17434895
]
László Bodor commented on HIVE-25637:
-
Finally, it turned out we need 2 changes:
1. Consider all TableDesc objects for kafka credentials which are present in
MapWork. This is because HIVE-23408 has taken care of TableDescs of the source
table, but while inserting, the source table is a dummy table, explain is like:
{code}
| Stage: Stage-2 |
| Tez|
| DagId: hive_20211026130429_4c237196-f3ef-4edb-8cc5-584d63f8b03d:4 |
| DagName: hive_20211026130429_4c237196-f3ef-4edb-8cc5-584d63f8b03d:4 |
| Vertices:|
| Map 1 |
| Map Operator Tree: |
| TableScan |
| alias: _dummy_table |
| Row Limit Per Split: 1 |
| Statistics: Num rows: 1 Data size: 10 Basic stats: COMPLETE
Column stats: COMPLETE |
| Select Operator |
| expressions: array(const
struct(null,'comment',0,1,2,3,null,null,null,null)) (type:
array>)
|
| outputColumnNames: _col0 |
| Statistics: Num rows: 1 Data size: 48 Basic stats:
COMPLETE Column stats: COMPLETE |
| UDTF Operator |
| Statistics: Num rows: 1 Data size: 48 Basic stats:
COMPLETE Column stats: COMPLETE |
| function name: inline|
| Select Operator |
| expressions: null (type: timestamp), col2 (type:
string), UDFToBoolean(col3) (type: boolean), col4 (type: int), UDFToLong(col5)
(type: bigint), UDFToDouble(col6) (type: double), null (type: binary), null
(type: int), null (type: bigint), null (type: bigint) |
| outputColumnNames: _col0, _col1, _col2, _col3, _col4,
_col5, _col6, _col7, _col8, _col9 |
| Statistics: Num rows: 1 Data size: 120 Basic stats:
COMPLETE Column stats: COMPLETE |
| File Output Operator |
| compressed: false|
| Statistics: Num rows: 1 Data size: 120 Basic stats:
COMPLETE Column stats: COMPLETE |
| table: |
| input format:
org.apache.hadoop.hive.kafka.KafkaInputFormat |
| output format:
org.apache.hadoop.hive.kafka.KafkaOutputFormat |
| serde: org.apache.hadoop.hive.kafka.KafkaSerDe |
| name: default.kafka_table |
||
{code}
2. merge credentials in TezProcessor to make them available for Hive operators,
because there were no tokens present in below codepath -> this caused a
fallback to GSSAPI (kerberos auth) instead of token based auth in KafkaUtils
{code}
at
org.apache.hadoop.hive.kafka.KafkaUtils.addKerberosJaasConf(KafkaUtils.java:362)
at
org.apache.hadoop.hive.kafka.KafkaUtils.producerProperties(KafkaUtils.java:230)
at
org.apache.hadoop.hive.kafka.KafkaOutputFormat.getHiveRecordWriter(KafkaOutputFormat.java:56)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:294)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:279)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:872)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:823)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1004)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670764
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:31
Start Date: 27/Oct/21 14:31
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737532309
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestDatabaseHandler.java
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.qoption;
+
+import org.apache.hadoop.hive.ql.QTestUtil;
+import org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB;
+import org.apache.hadoop.hive.ql.externalDB.MSSQLServer;
+import org.apache.hadoop.hive.ql.externalDB.MariaDB;
+import org.apache.hadoop.hive.ql.externalDB.MySQLExternalDB;
+import org.apache.hadoop.hive.ql.externalDB.Oracle;
+import org.apache.hadoop.hive.ql.externalDB.PostgresExternalDB;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+
+/**
+ * An option handler for spinning (resp. stopping) databases before (resp.
after) running a test.
+ *
+ * Syntax: qt:database:DatabaseType:path_to_init_script
+ *
+ * The database type ({@link DatabaseType}) is obligatory argument. The
initialization script can be omitted.
+ *
+ * Current limitations:
+ *
+ * Only one test/file per run
+ * Does not support parallel execution
+ * Cannot instantiate more than one database of the same type per
test
+ *
+ */
+public class QTestDatabaseHandler implements QTestOptionHandler {
+ private static final Logger LOG =
LoggerFactory.getLogger(QTestDatabaseHandler.class);
+
+ private enum DatabaseType {
+POSTGRES {
+ @Override
+ AbstractExternalDB create() {
+return new PostgresExternalDB();
+ }
+}, MYSQL {
+ @Override
+ AbstractExternalDB create() {
+return new MySQLExternalDB();
+ }
+}, MARIADB {
+ @Override
+ AbstractExternalDB create() {
+return new MariaDB();
+ }
+}, MSSQL {
+ @Override
+ AbstractExternalDB create() {
+return new MSSQLServer();
+ }
+}, ORACLE {
+ @Override
+ AbstractExternalDB create() {
+return new Oracle();
+ }
+};
+
+abstract AbstractExternalDB create();
+ }
+
+ private final Map databaseToScript = new
EnumMap<>(DatabaseType.class);
+
+ @Override
+ public void processArguments(String arguments) {
+String[] args = arguments.split(":");
+if (args.length == 0) {
+ throw new IllegalArgumentException("No arguments provided");
+}
+if (args.length > 2) {
+ throw new IllegalArgumentException(
+ "Too many arguments. Expected {dbtype:script}. Found: " +
Arrays.toString(args));
+}
+DatabaseType dbType = DatabaseType.valueOf(args[0].toUpperCase());
+String initScript = args.length == 2 ? args[1] : "";
+if (databaseToScript.containsKey(dbType)) {
+ throw new IllegalArgumentException(dbType + " database is already
defined in this file.");
+}
+databaseToScript.put(dbType, initScript);
+ }
+
+ @Override
+ public void beforeTest(QTestUtil qt) throws Exception {
+if (databaseToScript.isEmpty()) {
+ return;
+}
+for (Map.Entry dbEntry :
databaseToScript.entrySet()) {
+ String scriptsDir = QTestUtil.getScriptsDir(qt.getConf());
+ Path dbScript = Paths.get(scriptsDir, dbEntry.getValue());
+ AbstractExternalDB db = dbEntry.getKey().create();
Review comment:
Didn't get that, can you elaborate a bit more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670761
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:29
Start Date: 27/Oct/21 14:29
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737529431
##
File path: data/scripts/q_test_state_table.sql
##
@@ -0,0 +1,18 @@
+create table state
+(
+namevarchar(255) not null,
+country int not null
+);
+
+insert into state (name, country)
+values ('Maharashtra', 1);
+insert into state (name, country)
+values ('Madhya Pradesh', 1);
+insert into state (name, country)
+values ('Moscow', 3);
+insert into state (name, country)
+values ('Something', 4);
+insert into state (name, country)
+values ('Florida', 4);
+insert into state (name, country)
+values ('Texas', 4);
Review comment:
Fixed in
https://github.com/apache/hive/pull/2742/commits/f2ec2c66740b380ba3af67512dd6343fe6c7323e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670761)
Time Spent: 1.5h (was: 1h 20m)
> Setup JDBC databases in tests via QT options
>
>
> Key: HIVE-25594
> URL: https://issues.apache.org/jira/browse/HIVE-25594
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The goal of this jira is to add a new QT option for setting up JDBC DBMS and
> using it in qtests which need a JDBC endpoint up and running. It can be used
> in tests with external JDBC tables, connectors, etc.
> A sample file using the proposed option ({{qt:database}}) is shown below.
> {code:sql}
> --!qt:database:postgres:init_sript_1234.sql:cleanup_script_1234.sql
> CREATE EXTERNAL TABLE country (name varchar(80))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/qtestDB",
> "hive.sql.dbcp.username" = "qtestuser",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country");
> EXPLAIN CBO SELECT COUNT(*) from country;
> SELECT COUNT(*) from country;
> {code}
> This builds upon HIVE-25423 but proposes to use JDBC datasources without the
> need for a using a specific CLI driver. Furthermore, the proposed QT option
> syntax allows using customised init/cleanup scripts for the JDBC datasource
> per test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670760&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670760
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:27
Start Date: 27/Oct/21 14:27
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737527955
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##
@@ -193,96 +147,62 @@ public final String getContainerHostAddress() {
}
}
-public abstract void setJdbcUrl(String hostAddress);
-
-public abstract void setJdbcDriver();
-
-public abstract String getDockerImageName();
-
-public abstract String[] getDockerAdditionalArgs();
-
-public abstract boolean isContainerReady(ProcessResults pr);
+/**
+ * Return the name of the root user.
+ *
+ * Override the method if the name of the root user must be different than
the default.
+ */
+protected String getRootUser() {
+return "qtestuser";
+}
-protected String[] buildArray(String... strs) {
-return strs;
+/**
+ * Return the password of the root user.
+ *
+ * Override the method if the password must be different than the default.
+ */
+protected String getRootPassword() {
+return "qtestpassword";
}
+
+protected abstract String getJdbcUrl();
-public Connection getConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-try {
-LOG.info("external database connection URL:\t " + url);
-LOG.info("JDBC Driver :\t " + driver);
-LOG.info("external database connection User:\t " + userName);
-LOG.info("external database connection Password:\t " + password);
+protected abstract String getJdbcDriver();
-// load required JDBC driver
-Class.forName(driver);
+protected abstract String getDockerImageName();
-// Connect using the JDBC URL and user/password
-Connection conn = DriverManager.getConnection(url, userName,
password);
-return conn;
-} catch (SQLException e) {
-LOG.error("Failed to connect to external databse", e);
-throw new SQLException(e);
-} catch (ClassNotFoundException e) {
-LOG.error("Unable to find driver class", e);
-throw new ClassNotFoundException("Unable to find driver class");
-}
-}
+protected abstract String[] getDockerAdditionalArgs();
-public void testConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-Connection conn = getConnectionToExternalDB();
-try {
-conn.close();
-} catch (SQLException e) {
-LOG.error("Failed to close external database connection", e);
-}
-}
+protected abstract boolean isContainerReady(ProcessResults pr);
-protected String[] SQLLineCmdBuild(String sqlScriptFile) throws
IOException {
-return new String[] {"-u", url,
-"-d", driver,
-"-n", userName,
-"-p", password,
+private String[] SQLLineCmdBuild(String sqlScriptFile) {
+return new String[] {"-u", getJdbcUrl(),
+"-d", getJdbcDriver(),
+"-n", getRootUser(),
+"-p", getRootPassword(),
"--isolation=TRANSACTION_READ_COMMITTED",
"-f", sqlScriptFile};
}
-protected void execSql(String sqlScriptFile) throws IOException {
-// run the script using SqlLine
-SqlLine sqlLine = new SqlLine();
-ByteArrayOutputStream outputForLog = null;
-OutputStream out;
-if (LOG.isDebugEnabled()) {
-out = outputForLog = new ByteArrayOutputStream();
-} else {
-out = new NullOutputStream();
+public void execute(String script) throws IOException, SQLException,
ClassNotFoundException {
+// Test we can connect to database
+Class.forName(getJdbcDriver());
+try (Connection ignored = DriverManager.getConnection(getJdbcUrl(),
getRootUser(), getRootPassword())) {
+LOG.info("Successfully connected to {} with user {} and password
{}", getJdbcUrl(), getRootUser(), getRootPassword());
}
+LOG.info("Starting {} initialization", getClass().getSimpleName());
+SqlLine sqlLine = new SqlLine();
+ByteArrayOutputStream out = new ByteArrayOutputStream();
sqlLine.setOutputStream(new PrintStream(out));
+sqlLine.setErrorStream(new PrintStream(out));
System.
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670759&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670759
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:25
Start Date: 27/Oct/21 14:25
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737525094
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##
@@ -71,27 +59,8 @@ public ProcessResults(String stdout, String stderr, int rc) {
}
}
-public static AbstractExternalDB initalizeExternalDB(String
externalDBType) throws IOException {
-AbstractExternalDB abstractExternalDB;
-switch (externalDBType) {
-case "mysql":
-abstractExternalDB = new MySQLExternalDB();
-break;
-case "postgres":
-abstractExternalDB = new PostgresExternalDB();
-break;
-default:
-throw new IOException("unsupported external database type " +
externalDBType);
-}
-return abstractExternalDB;
-}
-
-public AbstractExternalDB(String externalDBType) {
-this.externalDBType = externalDBType;
-}
-
-protected String getDockerContainerName() {
-return String.format("qtestExternalDB-%s", externalDBType);
+private final String getDockerContainerName() {
Review comment:
There are indeed many similarities with the classes in `DatabaseRule`
although the latter contain additional code related to setting up and using a
Hive metastore. I do agree that a refactoring needs to be done but maybe it
would be better to leave it out of this change and treat it in a follow-up.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670759)
Time Spent: 1h 10m (was: 1h)
> Setup JDBC databases in tests via QT options
>
>
> Key: HIVE-25594
> URL: https://issues.apache.org/jira/browse/HIVE-25594
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The goal of this jira is to add a new QT option for setting up JDBC DBMS and
> using it in qtests which need a JDBC endpoint up and running. It can be used
> in tests with external JDBC tables, connectors, etc.
> A sample file using the proposed option ({{qt:database}}) is shown below.
> {code:sql}
> --!qt:database:postgres:init_sript_1234.sql:cleanup_script_1234.sql
> CREATE EXTERNAL TABLE country (name varchar(80))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/qtestDB",
> "hive.sql.dbcp.username" = "qtestuser",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country");
> EXPLAIN CBO SELECT COUNT(*) from country;
> SELECT COUNT(*) from country;
> {code}
> This builds upon HIVE-25423 but proposes to use JDBC datasources without the
> need for a using a specific CLI driver. Furthermore, the proposed QT option
> syntax allows using customised init/cleanup scripts for the JDBC datasource
> per test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25630) Transformer fixes
[ https://issues.apache.org/jira/browse/HIVE-25630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-25630. - Fix Version/s: 4.0.0 Resolution: Fixed merged into master. Thank you [~kkasa] for revieweing the changes! > Transformer fixes > - > > Key: HIVE-25630 > URL: https://issues.apache.org/jira/browse/HIVE-25630 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > there are some issues: > * AlreadyExistsException might be suppressed by the translator > * uppercase letter usage may cause problems for some clients > * add a way to suppress location checks for legacy clients -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25630) Transformer fixes
[ https://issues.apache.org/jira/browse/HIVE-25630?focusedWorklogId=670758&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670758 ] ASF GitHub Bot logged work on HIVE-25630: - Author: ASF GitHub Bot Created on: 27/Oct/21 14:22 Start Date: 27/Oct/21 14:22 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #2738: URL: https://github.com/apache/hive/pull/2738 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 670758) Time Spent: 40m (was: 0.5h) > Transformer fixes > - > > Key: HIVE-25630 > URL: https://issues.apache.org/jira/browse/HIVE-25630 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > there are some issues: > * AlreadyExistsException might be suppressed by the translator > * uppercase letter usage may cause problems for some clients > * add a way to suppress location checks for legacy clients -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670753&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670753
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:19
Start Date: 27/Oct/21 14:19
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737519278
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##
@@ -193,96 +147,62 @@ public final String getContainerHostAddress() {
}
}
-public abstract void setJdbcUrl(String hostAddress);
-
-public abstract void setJdbcDriver();
-
-public abstract String getDockerImageName();
-
-public abstract String[] getDockerAdditionalArgs();
-
-public abstract boolean isContainerReady(ProcessResults pr);
+/**
+ * Return the name of the root user.
+ *
+ * Override the method if the name of the root user must be different than
the default.
+ */
+protected String getRootUser() {
+return "qtestuser";
+}
-protected String[] buildArray(String... strs) {
-return strs;
+/**
+ * Return the password of the root user.
+ *
+ * Override the method if the password must be different than the default.
+ */
+protected String getRootPassword() {
+return "qtestpassword";
}
+
+protected abstract String getJdbcUrl();
-public Connection getConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-try {
-LOG.info("external database connection URL:\t " + url);
-LOG.info("JDBC Driver :\t " + driver);
-LOG.info("external database connection User:\t " + userName);
-LOG.info("external database connection Password:\t " + password);
+protected abstract String getJdbcDriver();
-// load required JDBC driver
-Class.forName(driver);
+protected abstract String getDockerImageName();
-// Connect using the JDBC URL and user/password
-Connection conn = DriverManager.getConnection(url, userName,
password);
-return conn;
-} catch (SQLException e) {
-LOG.error("Failed to connect to external databse", e);
-throw new SQLException(e);
-} catch (ClassNotFoundException e) {
-LOG.error("Unable to find driver class", e);
-throw new ClassNotFoundException("Unable to find driver class");
-}
-}
+protected abstract String[] getDockerAdditionalArgs();
-public void testConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-Connection conn = getConnectionToExternalDB();
-try {
-conn.close();
-} catch (SQLException e) {
-LOG.error("Failed to close external database connection", e);
-}
-}
+protected abstract boolean isContainerReady(ProcessResults pr);
-protected String[] SQLLineCmdBuild(String sqlScriptFile) throws
IOException {
-return new String[] {"-u", url,
-"-d", driver,
-"-n", userName,
-"-p", password,
+private String[] SQLLineCmdBuild(String sqlScriptFile) {
+return new String[] {"-u", getJdbcUrl(),
+"-d", getJdbcDriver(),
+"-n", getRootUser(),
+"-p", getRootPassword(),
"--isolation=TRANSACTION_READ_COMMITTED",
"-f", sqlScriptFile};
}
-protected void execSql(String sqlScriptFile) throws IOException {
-// run the script using SqlLine
-SqlLine sqlLine = new SqlLine();
-ByteArrayOutputStream outputForLog = null;
-OutputStream out;
-if (LOG.isDebugEnabled()) {
-out = outputForLog = new ByteArrayOutputStream();
-} else {
-out = new NullOutputStream();
+public void execute(String script) throws IOException, SQLException,
ClassNotFoundException {
+// Test we can connect to database
+Class.forName(getJdbcDriver());
+try (Connection ignored = DriverManager.getConnection(getJdbcUrl(),
getRootUser(), getRootPassword())) {
+LOG.info("Successfully connected to {} with user {} and password
{}", getJdbcUrl(), getRootUser(), getRootPassword());
}
+LOG.info("Starting {} initialization", getClass().getSimpleName());
+SqlLine sqlLine = new SqlLine();
+ByteArrayOutputStream out = new ByteArrayOutputStream();
sqlLine.setOutputStream(new PrintStream(out));
+sqlLine.setErrorStream(new PrintStream(out));
System.
[jira] [Work logged] (HIVE-25655) Remove ElapsedTimeLoggingWrapper from tests
[
https://issues.apache.org/jira/browse/HIVE-25655?focusedWorklogId=670751&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670751
]
ASF GitHub Bot logged work on HIVE-25655:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:17
Start Date: 27/Oct/21 14:17
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request #2755:
URL: https://github.com/apache/hive/pull/2755
### What changes were proposed in this pull request?
Remove ElapsedTimeLoggingWrapper and its usages since the benefit of logging
times in before/after methods is subtle.
### Why are the changes needed?
1. Improve code readability
2. Reduce maintenance overhead
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Refactoring so no new tests needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670751)
Remaining Estimate: 0h
Time Spent: 10m
> Remove ElapsedTimeLoggingWrapper from tests
> ---
>
> Key: HIVE-25655
> URL: https://issues.apache.org/jira/browse/HIVE-25655
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The
> [ElapsedTimeLoggingWrapper|https://github.com/apache/hive/blob/f749ef2af27638914984c183bcfa213920f5cdd9/itests/util/src/main/java/org/apache/hadoop/hive/util/ElapsedTimeLoggingWrapper.java]
> introduced in HIVE-14625 is used by the [CoreCliDriver|#L68] to execute,
> measure, and display the time spend on some operations during the execution
> of {{@Before/@After}} methods.
> The benefit of logging the elapsed time for these methods is unclear. The
> time is usually rather short, especially compared to the actual time a query
> takes to run, so it is not an information which can be of much use.
> The enforced coding pattern for measuring and logging the time leads to
> boilerplate and makes the code harder to read and understand.
> {code:java}
> qt = new ElapsedTimeLoggingWrapper() {
> @Override
> public QTestUtil invokeInternal() throws Exception {
> return new QTestUtil(
> QTestArguments.QTestArgumentsBuilder.instance()
> .withOutDir(cliConfig.getResultsDir())
> .withLogDir(cliConfig.getLogDir())
> .withClusterType(miniMR)
> .withConfDir(hiveConfDir)
> .withInitScript(initScript)
> .withCleanupScript(cleanupScript)
> .withLlapIo(true)
> .withFsType(cliConfig.getFsType())
> .build());
> }
> }.invoke("QtestUtil instance created", LOG, true);
> {code}
> Moreover, the wrapper is not used consistently across drivers making results
> less uniform.
> The goal of this issue is to remove {{ElapsedTimeLoggingWrapper}} and its
> usages to improve code readability and maintenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25655) Remove ElapsedTimeLoggingWrapper from tests
[
https://issues.apache.org/jira/browse/HIVE-25655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25655:
--
Labels: pull-request-available (was: )
> Remove ElapsedTimeLoggingWrapper from tests
> ---
>
> Key: HIVE-25655
> URL: https://issues.apache.org/jira/browse/HIVE-25655
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The
> [ElapsedTimeLoggingWrapper|https://github.com/apache/hive/blob/f749ef2af27638914984c183bcfa213920f5cdd9/itests/util/src/main/java/org/apache/hadoop/hive/util/ElapsedTimeLoggingWrapper.java]
> introduced in HIVE-14625 is used by the [CoreCliDriver|#L68] to execute,
> measure, and display the time spend on some operations during the execution
> of {{@Before/@After}} methods.
> The benefit of logging the elapsed time for these methods is unclear. The
> time is usually rather short, especially compared to the actual time a query
> takes to run, so it is not an information which can be of much use.
> The enforced coding pattern for measuring and logging the time leads to
> boilerplate and makes the code harder to read and understand.
> {code:java}
> qt = new ElapsedTimeLoggingWrapper() {
> @Override
> public QTestUtil invokeInternal() throws Exception {
> return new QTestUtil(
> QTestArguments.QTestArgumentsBuilder.instance()
> .withOutDir(cliConfig.getResultsDir())
> .withLogDir(cliConfig.getLogDir())
> .withClusterType(miniMR)
> .withConfDir(hiveConfDir)
> .withInitScript(initScript)
> .withCleanupScript(cleanupScript)
> .withLlapIo(true)
> .withFsType(cliConfig.getFsType())
> .build());
> }
> }.invoke("QtestUtil instance created", LOG, true);
> {code}
> Moreover, the wrapper is not used consistently across drivers making results
> less uniform.
> The goal of this issue is to remove {{ElapsedTimeLoggingWrapper}} and its
> usages to improve code readability and maintenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670749&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670749
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:12
Start Date: 27/Oct/21 14:12
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737503807
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##
@@ -71,27 +59,8 @@ public ProcessResults(String stdout, String stderr, int rc) {
}
}
-public static AbstractExternalDB initalizeExternalDB(String
externalDBType) throws IOException {
-AbstractExternalDB abstractExternalDB;
-switch (externalDBType) {
-case "mysql":
-abstractExternalDB = new MySQLExternalDB();
-break;
-case "postgres":
-abstractExternalDB = new PostgresExternalDB();
-break;
-default:
-throw new IOException("unsupported external database type " +
externalDBType);
-}
-return abstractExternalDB;
-}
-
-public AbstractExternalDB(String externalDBType) {
-this.externalDBType = externalDBType;
-}
-
-protected String getDockerContainerName() {
-return String.format("qtestExternalDB-%s", externalDBType);
+private final String getDockerContainerName() {
Review comment:
how much these differ from (`
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
`)?
can't we reuse those classes?
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/MariaDB.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.externalDB;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class MariaDB extends AbstractExternalDB {
Review comment:
./standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/Mysql.java
?
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestDatabaseHandler.java
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.qoption;
+
+import org.apache.hadoop.hive.ql.QTestUtil;
+import org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB;
+import org.apache.hadoop.hive.ql.externalDB.MSSQLServer;
+import org.apache.hadoop.hive.ql.externalDB.MariaDB;
+import org.apache.hadoop.hive.ql.externalDB.MySQLExternalDB;
+import org.apache.hadoop.hive.ql.externalDB.Oracle;
+import org.apache.hadoop.hive.ql.externalDB.PostgresExternalDB;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+
+/**
+ * An option handler for spinning (resp. stopping) databases before (resp.
after) running a test.
+ *
+ * Syntax: qt:database:DatabaseType:path_to_init_script
+ *
+ * The database type ({@link DatabaseType}) is obligatory ar
[jira] [Assigned] (HIVE-25655) Remove ElapsedTimeLoggingWrapper from tests
[
https://issues.apache.org/jira/browse/HIVE-25655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-25655:
--
> Remove ElapsedTimeLoggingWrapper from tests
> ---
>
> Key: HIVE-25655
> URL: https://issues.apache.org/jira/browse/HIVE-25655
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> The
> [ElapsedTimeLoggingWrapper|https://github.com/apache/hive/blob/f749ef2af27638914984c183bcfa213920f5cdd9/itests/util/src/main/java/org/apache/hadoop/hive/util/ElapsedTimeLoggingWrapper.java]
> introduced in HIVE-14625 is used by the [CoreCliDriver|#L68] to execute,
> measure, and display the time spend on some operations during the execution
> of {{@Before/@After}} methods.
> The benefit of logging the elapsed time for these methods is unclear. The
> time is usually rather short, especially compared to the actual time a query
> takes to run, so it is not an information which can be of much use.
> The enforced coding pattern for measuring and logging the time leads to
> boilerplate and makes the code harder to read and understand.
> {code:java}
> qt = new ElapsedTimeLoggingWrapper() {
> @Override
> public QTestUtil invokeInternal() throws Exception {
> return new QTestUtil(
> QTestArguments.QTestArgumentsBuilder.instance()
> .withOutDir(cliConfig.getResultsDir())
> .withLogDir(cliConfig.getLogDir())
> .withClusterType(miniMR)
> .withConfDir(hiveConfDir)
> .withInitScript(initScript)
> .withCleanupScript(cleanupScript)
> .withLlapIo(true)
> .withFsType(cliConfig.getFsType())
> .build());
> }
> }.invoke("QtestUtil instance created", LOG, true);
> {code}
> Moreover, the wrapper is not used consistently across drivers making results
> less uniform.
> The goal of this issue is to remove {{ElapsedTimeLoggingWrapper}} and its
> usages to improve code readability and maintenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25655) Remove ElapsedTimeLoggingWrapper from tests
[
https://issues.apache.org/jira/browse/HIVE-25655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-25655:
---
Priority: Minor (was: Major)
> Remove ElapsedTimeLoggingWrapper from tests
> ---
>
> Key: HIVE-25655
> URL: https://issues.apache.org/jira/browse/HIVE-25655
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
>
> The
> [ElapsedTimeLoggingWrapper|https://github.com/apache/hive/blob/f749ef2af27638914984c183bcfa213920f5cdd9/itests/util/src/main/java/org/apache/hadoop/hive/util/ElapsedTimeLoggingWrapper.java]
> introduced in HIVE-14625 is used by the [CoreCliDriver|#L68] to execute,
> measure, and display the time spend on some operations during the execution
> of {{@Before/@After}} methods.
> The benefit of logging the elapsed time for these methods is unclear. The
> time is usually rather short, especially compared to the actual time a query
> takes to run, so it is not an information which can be of much use.
> The enforced coding pattern for measuring and logging the time leads to
> boilerplate and makes the code harder to read and understand.
> {code:java}
> qt = new ElapsedTimeLoggingWrapper() {
> @Override
> public QTestUtil invokeInternal() throws Exception {
> return new QTestUtil(
> QTestArguments.QTestArgumentsBuilder.instance()
> .withOutDir(cliConfig.getResultsDir())
> .withLogDir(cliConfig.getLogDir())
> .withClusterType(miniMR)
> .withConfDir(hiveConfDir)
> .withInitScript(initScript)
> .withCleanupScript(cleanupScript)
> .withLlapIo(true)
> .withFsType(cliConfig.getFsType())
> .build());
> }
> }.invoke("QtestUtil instance created", LOG, true);
> {code}
> Moreover, the wrapper is not used consistently across drivers making results
> less uniform.
> The goal of this issue is to remove {{ElapsedTimeLoggingWrapper}} and its
> usages to improve code readability and maintenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670745&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670745
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:02
Start Date: 27/Oct/21 14:02
Worklog Time Spent: 10m
Work Description: asolimando commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737501273
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##
@@ -193,96 +147,62 @@ public final String getContainerHostAddress() {
}
}
-public abstract void setJdbcUrl(String hostAddress);
-
-public abstract void setJdbcDriver();
-
-public abstract String getDockerImageName();
-
-public abstract String[] getDockerAdditionalArgs();
-
-public abstract boolean isContainerReady(ProcessResults pr);
+/**
+ * Return the name of the root user.
+ *
+ * Override the method if the name of the root user must be different than
the default.
+ */
+protected String getRootUser() {
+return "qtestuser";
+}
-protected String[] buildArray(String... strs) {
-return strs;
+/**
+ * Return the password of the root user.
+ *
+ * Override the method if the password must be different than the default.
+ */
+protected String getRootPassword() {
+return "qtestpassword";
}
+
+protected abstract String getJdbcUrl();
-public Connection getConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-try {
-LOG.info("external database connection URL:\t " + url);
-LOG.info("JDBC Driver :\t " + driver);
-LOG.info("external database connection User:\t " + userName);
-LOG.info("external database connection Password:\t " + password);
+protected abstract String getJdbcDriver();
-// load required JDBC driver
-Class.forName(driver);
+protected abstract String getDockerImageName();
-// Connect using the JDBC URL and user/password
-Connection conn = DriverManager.getConnection(url, userName,
password);
-return conn;
-} catch (SQLException e) {
-LOG.error("Failed to connect to external databse", e);
-throw new SQLException(e);
-} catch (ClassNotFoundException e) {
-LOG.error("Unable to find driver class", e);
-throw new ClassNotFoundException("Unable to find driver class");
-}
-}
+protected abstract String[] getDockerAdditionalArgs();
-public void testConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-Connection conn = getConnectionToExternalDB();
-try {
-conn.close();
-} catch (SQLException e) {
-LOG.error("Failed to close external database connection", e);
-}
-}
+protected abstract boolean isContainerReady(ProcessResults pr);
-protected String[] SQLLineCmdBuild(String sqlScriptFile) throws
IOException {
-return new String[] {"-u", url,
-"-d", driver,
-"-n", userName,
-"-p", password,
+private String[] SQLLineCmdBuild(String sqlScriptFile) {
+return new String[] {"-u", getJdbcUrl(),
+"-d", getJdbcDriver(),
+"-n", getRootUser(),
+"-p", getRootPassword(),
"--isolation=TRANSACTION_READ_COMMITTED",
"-f", sqlScriptFile};
}
-protected void execSql(String sqlScriptFile) throws IOException {
-// run the script using SqlLine
-SqlLine sqlLine = new SqlLine();
-ByteArrayOutputStream outputForLog = null;
-OutputStream out;
-if (LOG.isDebugEnabled()) {
-out = outputForLog = new ByteArrayOutputStream();
-} else {
-out = new NullOutputStream();
+public void execute(String script) throws IOException, SQLException,
ClassNotFoundException {
+// Test we can connect to database
+Class.forName(getJdbcDriver());
+try (Connection ignored = DriverManager.getConnection(getJdbcUrl(),
getRootUser(), getRootPassword())) {
+LOG.info("Successfully connected to {} with user {} and password
{}", getJdbcUrl(), getRootUser(), getRootPassword());
}
+LOG.info("Starting {} initialization", getClass().getSimpleName());
+SqlLine sqlLine = new SqlLine();
+ByteArrayOutputStream out = new ByteArrayOutputStream();
sqlLine.setOutputStream(new PrintStream(out));
+sqlLine.setErrorStream(new PrintStream(out));
Syst
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670744
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 14:01
Start Date: 27/Oct/21 14:01
Worklog Time Spent: 10m
Work Description: asolimando commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737500744
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##
@@ -193,96 +147,62 @@ public final String getContainerHostAddress() {
}
}
-public abstract void setJdbcUrl(String hostAddress);
-
-public abstract void setJdbcDriver();
-
-public abstract String getDockerImageName();
-
-public abstract String[] getDockerAdditionalArgs();
-
-public abstract boolean isContainerReady(ProcessResults pr);
+/**
+ * Return the name of the root user.
+ *
+ * Override the method if the name of the root user must be different than
the default.
+ */
+protected String getRootUser() {
+return "qtestuser";
+}
-protected String[] buildArray(String... strs) {
-return strs;
+/**
+ * Return the password of the root user.
+ *
+ * Override the method if the password must be different than the default.
+ */
+protected String getRootPassword() {
+return "qtestpassword";
}
+
+protected abstract String getJdbcUrl();
-public Connection getConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-try {
-LOG.info("external database connection URL:\t " + url);
-LOG.info("JDBC Driver :\t " + driver);
-LOG.info("external database connection User:\t " + userName);
-LOG.info("external database connection Password:\t " + password);
+protected abstract String getJdbcDriver();
-// load required JDBC driver
-Class.forName(driver);
+protected abstract String getDockerImageName();
-// Connect using the JDBC URL and user/password
-Connection conn = DriverManager.getConnection(url, userName,
password);
-return conn;
-} catch (SQLException e) {
-LOG.error("Failed to connect to external databse", e);
-throw new SQLException(e);
-} catch (ClassNotFoundException e) {
-LOG.error("Unable to find driver class", e);
-throw new ClassNotFoundException("Unable to find driver class");
-}
-}
+protected abstract String[] getDockerAdditionalArgs();
-public void testConnectionToExternalDB() throws SQLException,
ClassNotFoundException {
-Connection conn = getConnectionToExternalDB();
-try {
-conn.close();
-} catch (SQLException e) {
-LOG.error("Failed to close external database connection", e);
-}
-}
+protected abstract boolean isContainerReady(ProcessResults pr);
-protected String[] SQLLineCmdBuild(String sqlScriptFile) throws
IOException {
-return new String[] {"-u", url,
-"-d", driver,
-"-n", userName,
-"-p", password,
+private String[] SQLLineCmdBuild(String sqlScriptFile) {
+return new String[] {"-u", getJdbcUrl(),
+"-d", getJdbcDriver(),
+"-n", getRootUser(),
+"-p", getRootPassword(),
"--isolation=TRANSACTION_READ_COMMITTED",
"-f", sqlScriptFile};
}
-protected void execSql(String sqlScriptFile) throws IOException {
-// run the script using SqlLine
-SqlLine sqlLine = new SqlLine();
-ByteArrayOutputStream outputForLog = null;
-OutputStream out;
-if (LOG.isDebugEnabled()) {
-out = outputForLog = new ByteArrayOutputStream();
-} else {
-out = new NullOutputStream();
+public void execute(String script) throws IOException, SQLException,
ClassNotFoundException {
+// Test we can connect to database
+Class.forName(getJdbcDriver());
+try (Connection ignored = DriverManager.getConnection(getJdbcUrl(),
getRootUser(), getRootPassword())) {
+LOG.info("Successfully connected to {} with user {} and password
{}", getJdbcUrl(), getRootUser(), getRootPassword());
}
+LOG.info("Starting {} initialization", getClass().getSimpleName());
+SqlLine sqlLine = new SqlLine();
+ByteArrayOutputStream out = new ByteArrayOutputStream();
sqlLine.setOutputStream(new PrintStream(out));
+sqlLine.setErrorStream(new PrintStream(out));
Syst
[jira] [Work logged] (HIVE-25594) Setup JDBC databases in tests via QT options
[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=670742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670742
]
ASF GitHub Bot logged work on HIVE-25594:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 13:55
Start Date: 27/Oct/21 13:55
Worklog Time Spent: 10m
Work Description: asolimando commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r737494195
##
File path: data/scripts/q_test_state_table.sql
##
@@ -0,0 +1,18 @@
+create table state
+(
+namevarchar(255) not null,
+country int not null
+);
+
+insert into state (name, country)
+values ('Maharashtra', 1);
+insert into state (name, country)
+values ('Madhya Pradesh', 1);
+insert into state (name, country)
+values ('Moscow', 3);
+insert into state (name, country)
+values ('Something', 4);
+insert into state (name, country)
+values ('Florida', 4);
+insert into state (name, country)
+values ('Texas', 4);
Review comment:
Newline is missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670742)
Time Spent: 20m (was: 10m)
> Setup JDBC databases in tests via QT options
>
>
> Key: HIVE-25594
> URL: https://issues.apache.org/jira/browse/HIVE-25594
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The goal of this jira is to add a new QT option for setting up JDBC DBMS and
> using it in qtests which need a JDBC endpoint up and running. It can be used
> in tests with external JDBC tables, connectors, etc.
> A sample file using the proposed option ({{qt:database}}) is shown below.
> {code:sql}
> --!qt:database:postgres:init_sript_1234.sql:cleanup_script_1234.sql
> CREATE EXTERNAL TABLE country (name varchar(80))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/qtestDB",
> "hive.sql.dbcp.username" = "qtestuser",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country");
> EXPLAIN CBO SELECT COUNT(*) from country;
> SELECT COUNT(*) from country;
> {code}
> This builds upon HIVE-25423 but proposes to use JDBC datasources without the
> need for a using a specific CLI driver. Furthermore, the proposed QT option
> syntax allows using customised init/cleanup scripts for the JDBC datasource
> per test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25642) Log a warning if multiple Compaction Worker versions are running compactions
[
https://issues.apache.org/jira/browse/HIVE-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viktor Csomor resolved HIVE-25642.
--
Fix Version/s: 4.0.0
Release Note: PR: https://github.com/apache/hive/pull/2743
Resolution: Fixed
> Log a warning if multiple Compaction Worker versions are running compactions
>
>
> Key: HIVE-25642
> URL: https://issues.apache.org/jira/browse/HIVE-25642
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Log a warning if multiple versions of a Compaction Workers are running
> compactions.
> The start time of the individual HMS services are not stored at the moment,
> however this information could proved a good baseline of detecting multiple
> Worker versions.
> Due to the lack of this information we can check periodically in the past N
> hours to detect the versions.
> The N hours can be configured by
> {{metastore.compactor.worker.detect_multiple_versions.threshold}} property.
> This periodical check only make sense if the Compaction are enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25642) Log a warning if multiple Compaction Worker versions are running compactions
[
https://issues.apache.org/jira/browse/HIVE-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434836#comment-17434836
]
László Pintér commented on HIVE-25642:
--
Merged into master. Thanks for the patch, [~vcsomor]!
> Log a warning if multiple Compaction Worker versions are running compactions
>
>
> Key: HIVE-25642
> URL: https://issues.apache.org/jira/browse/HIVE-25642
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Log a warning if multiple versions of a Compaction Workers are running
> compactions.
> The start time of the individual HMS services are not stored at the moment,
> however this information could proved a good baseline of detecting multiple
> Worker versions.
> Due to the lack of this information we can check periodically in the past N
> hours to detect the versions.
> The N hours can be configured by
> {{metastore.compactor.worker.detect_multiple_versions.threshold}} property.
> This periodical check only make sense if the Compaction are enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25642) Log a warning if multiple Compaction Worker versions are running compactions
[
https://issues.apache.org/jira/browse/HIVE-25642?focusedWorklogId=670685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670685
]
ASF GitHub Bot logged work on HIVE-25642:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 12:28
Start Date: 27/Oct/21 12:28
Worklog Time Spent: 10m
Work Description: lcspinter merged pull request #2743:
URL: https://github.com/apache/hive/pull/2743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670685)
Time Spent: 40m (was: 0.5h)
> Log a warning if multiple Compaction Worker versions are running compactions
>
>
> Key: HIVE-25642
> URL: https://issues.apache.org/jira/browse/HIVE-25642
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Log a warning if multiple versions of a Compaction Workers are running
> compactions.
> The start time of the individual HMS services are not stored at the moment,
> however this information could proved a good baseline of detecting multiple
> Worker versions.
> Due to the lack of this information we can check periodically in the past N
> hours to detect the versions.
> The N hours can be configured by
> {{metastore.compactor.worker.detect_multiple_versions.threshold}} property.
> This periodical check only make sense if the Compaction are enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25634) Eclipse compiler bumps into AIOBE during ObjectStore compilation
[ https://issues.apache.org/jira/browse/HIVE-25634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434787#comment-17434787 ] Zhihua Deng commented on HIVE-25634: [~kgyrtkirk] I proposed a quick fix for this, Would you mind taking a look if have some seconds? Thank you! > Eclipse compiler bumps into AIOBE during ObjectStore compilation > > > Key: HIVE-25634 > URL: https://issues.apache.org/jira/browse/HIVE-25634 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > this issue seem to have started appearing after HIVE-23633 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25634) Eclipse compiler bumps into AIOBE during ObjectStore compilation
[ https://issues.apache.org/jira/browse/HIVE-25634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25634: -- Labels: pull-request-available (was: ) > Eclipse compiler bumps into AIOBE during ObjectStore compilation > > > Key: HIVE-25634 > URL: https://issues.apache.org/jira/browse/HIVE-25634 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > this issue seem to have started appearing after HIVE-23633 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25634) Eclipse compiler bumps into AIOBE during ObjectStore compilation
[ https://issues.apache.org/jira/browse/HIVE-25634?focusedWorklogId=670641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670641 ] ASF GitHub Bot logged work on HIVE-25634: - Author: ASF GitHub Bot Created on: 27/Oct/21 11:01 Start Date: 27/Oct/21 11:01 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2754: URL: https://github.com/apache/hive/pull/2754 …ilation ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 670641) Remaining Estimate: 0h Time Spent: 10m > Eclipse compiler bumps into AIOBE during ObjectStore compilation > > > Key: HIVE-25634 > URL: https://issues.apache.org/jira/browse/HIVE-25634 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > this issue seem to have started appearing after HIVE-23633 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25653) Precision problem in STD, STDDDEV_SAMP,STDDEV_POP
[
https://issues.apache.org/jira/browse/HIVE-25653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Sharma updated HIVE-25653:
-
Description:
Description
*Script *-
create table test ( col1 int );
insert into values
('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72');
select STDDEV_SAMP(col1) AS STDDEV_6M , STDDEV(col1) as STDDEV
,STDDEV_POP(col1) as STDDEV_POP from test;
*Result *-
STDDDEV_SAMPSTDDEV STDDEV_POP
5.940794514955821E-13 5.42317860890711E-13 5.42317860890711E-13
*Expected *-
STDDDEV_SAMPSTDDEV STDDEV_POP
0 0
0
was:
Description
> Precision problem in STD, STDDDEV_SAMP,STDDEV_POP
> -
>
> Key: HIVE-25653
> URL: https://issues.apache.org/jira/browse/HIVE-25653
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
>
> Description
> *Script *-
> create table test ( col1 int );
> insert into values
> ('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72');
> select STDDEV_SAMP(col1) AS STDDEV_6M , STDDEV(col1) as STDDEV
> ,STDDEV_POP(col1) as STDDEV_POP from test;
> *Result *-
> STDDDEV_SAMPSTDDEV
> STDDEV_POP
> 5.940794514955821E-13 5.42317860890711E-13 5.42317860890711E-13
> *Expected *-
> STDDDEV_SAMPSTDDEV
> STDDEV_POP
> 0 0
>0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25653) Precision problem in STD, STDDDEV_SAMP,STDDEV_POP
[
https://issues.apache.org/jira/browse/HIVE-25653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Sharma updated HIVE-25653:
-
Description:
Description
*Script*-
create table test ( col1 int );
insert into values
('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72');
select STDDEV_SAMP(col1) AS STDDEV_6M , STDDEV(col1) as STDDEV
,STDDEV_POP(col1) as STDDEV_POP from test;
*Result*-
STDDDEV_SAMPSTDDEV STDDEV_POP
5.940794514955821E-13 5.42317860890711E-13 5.42317860890711E-13
*Expected*-
STDDDEV_SAMPSTDDEV STDDEV_POP
0 0
0
was:
Description
*Script *-
create table test ( col1 int );
insert into values
('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72');
select STDDEV_SAMP(col1) AS STDDEV_6M , STDDEV(col1) as STDDEV
,STDDEV_POP(col1) as STDDEV_POP from test;
*Result *-
STDDDEV_SAMPSTDDEV STDDEV_POP
5.940794514955821E-13 5.42317860890711E-13 5.42317860890711E-13
*Expected *-
STDDDEV_SAMPSTDDEV STDDEV_POP
0 0
0
> Precision problem in STD, STDDDEV_SAMP,STDDEV_POP
> -
>
> Key: HIVE-25653
> URL: https://issues.apache.org/jira/browse/HIVE-25653
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
>
> Description
> *Script*-
> create table test ( col1 int );
> insert into values
> ('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72'),('10230.72');
> select STDDEV_SAMP(col1) AS STDDEV_6M , STDDEV(col1) as STDDEV
> ,STDDEV_POP(col1) as STDDEV_POP from test;
> *Result*-
> STDDDEV_SAMPSTDDEV
> STDDEV_POP
> 5.940794514955821E-13 5.42317860890711E-13 5.42317860890711E-13
> *Expected*-
> STDDDEV_SAMPSTDDEV
> STDDEV_POP
> 0 0
>0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25653) Precision problem in STD, STDDDEV_SAMP,STDDEV_POP
[ https://issues.apache.org/jira/browse/HIVE-25653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma reassigned HIVE-25653: > Precision problem in STD, STDDDEV_SAMP,STDDEV_POP > - > > Key: HIVE-25653 > URL: https://issues.apache.org/jira/browse/HIVE-25653 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > > Description -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25600) Compaction job creates redundant base/delta folder within base/delta folder
[
https://issues.apache.org/jira/browse/HIVE-25600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434723#comment-17434723
]
Sankar Hariappan commented on HIVE-25600:
-
Thanks [~gupta.nikhil0007]! Patch merged to branch-3.1.
> Compaction job creates redundant base/delta folder within base/delta folder
> ---
>
> Key: HIVE-25600
> URL: https://issues.apache.org/jira/browse/HIVE-25600
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 3.1.2
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Minor
> Labels: ACID, Compaction, pull-request-available
> Fix For: 3.1.3
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {noformat}
> Hive table 'aa.bb_item' is corrupt. Found sub-directory
> 'abfs://aabb-hive-d...@gen2hiveaabb.dfs.core.windows.net/prod-data/aa.db/bb_item/part_created_on=202105/base_0004042/base_0004042'
> in bucket directory for partition: part_created_on=202105
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:543)
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:325)
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:254)
> at io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
> at io.prestosql.$gen.Presto_34720210615_143054_2.run(Unknown Source)
> at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829);{noformat}
> Why it happens:
> Multiple compaction jobs for the same transactions can be triggered if the
> HMS gets restarted and the MR job is still in progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25600) Compaction job creates redundant base/delta folder within base/delta folder
[
https://issues.apache.org/jira/browse/HIVE-25600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan resolved HIVE-25600.
-
Target Version/s: 3.1.3
Resolution: Fixed
> Compaction job creates redundant base/delta folder within base/delta folder
> ---
>
> Key: HIVE-25600
> URL: https://issues.apache.org/jira/browse/HIVE-25600
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 3.1.2
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Minor
> Labels: ACID, Compaction, pull-request-available
> Fix For: 3.1.3
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {noformat}
> Hive table 'aa.bb_item' is corrupt. Found sub-directory
> 'abfs://aabb-hive-d...@gen2hiveaabb.dfs.core.windows.net/prod-data/aa.db/bb_item/part_created_on=202105/base_0004042/base_0004042'
> in bucket directory for partition: part_created_on=202105
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:543)
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:325)
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:254)
> at io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
> at io.prestosql.$gen.Presto_34720210615_143054_2.run(Unknown Source)
> at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829);{noformat}
> Why it happens:
> Multiple compaction jobs for the same transactions can be triggered if the
> HMS gets restarted and the MR job is still in progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25600) Compaction job creates redundant base/delta folder within base/delta folder
[
https://issues.apache.org/jira/browse/HIVE-25600?focusedWorklogId=670541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670541
]
ASF GitHub Bot logged work on HIVE-25600:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 08:06
Start Date: 27/Oct/21 08:06
Worklog Time Spent: 10m
Work Description: sankarh merged pull request #2705:
URL: https://github.com/apache/hive/pull/2705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670541)
Time Spent: 1h (was: 50m)
> Compaction job creates redundant base/delta folder within base/delta folder
> ---
>
> Key: HIVE-25600
> URL: https://issues.apache.org/jira/browse/HIVE-25600
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 3.1.2
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Minor
> Labels: ACID, Compaction, pull-request-available
> Fix For: 3.1.3
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {noformat}
> Hive table 'aa.bb_item' is corrupt. Found sub-directory
> 'abfs://aabb-hive-d...@gen2hiveaabb.dfs.core.windows.net/prod-data/aa.db/bb_item/part_created_on=202105/base_0004042/base_0004042'
> in bucket directory for partition: part_created_on=202105
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:543)
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:325)
> at
> io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:254)
> at io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
> at io.prestosql.$gen.Presto_34720210615_143054_2.run(Unknown Source)
> at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829);{noformat}
> Why it happens:
> Multiple compaction jobs for the same transactions can be triggered if the
> HMS gets restarted and the MR job is still in progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25637) Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
[
https://issues.apache.org/jira/browse/HIVE-25637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25637:
--
Labels: pull-request-available (was: )
> Hive on Tez: inserting data failing into the non native hive external table
> managed by kafka storage handler
> -
>
> Key: HIVE-25637
> URL: https://issues.apache.org/jira/browse/HIVE-25637
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is the followup for HIVE-23408, repro is below:
> {code}
> CREATE EXTERNAL TABLE `kafka_table`(
> `timestamp` timestamp COMMENT 'from deserializer',
> `page` string COMMENT 'from deserializer',
> `newpage` boolean COMMENT 'from deserializer',
> `added` int COMMENT 'from deserializer',
> `deleted` bigint COMMENT 'from deserializer',
> `delta` double COMMENT 'from deserializer')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.kafka.KafkaSerDe'
> STORED BY
> 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> WITH SERDEPROPERTIES (
> 'serialization.format'='1')
> LOCATION
>
> 'hdfs://lbodorkafkaunsec-2.lbodorkafkaunsec.root.hwx.site:8020/warehouse/tablespace/external/hive/kafka_table'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'hive.kafka.max.retries'='6',
> 'hive.kafka.metadata.poll.timeout.ms'='3',
> 'hive.kafka.optimistic.commit'='false',
> 'hive.kafka.poll.timeout.ms'='5000',
>
> 'kafka.bootstrap.servers'='lbodorkafkaunsec-1.lbodorkafkaunsec.root.hwx.site:9092,lbodorkafkaunsec-2.lbodorkafkaunsec.root.hwx.site:9092,lbodorkafkaunsec-3.lbodorkafkaunsec.root.hwx.site:9092',
> 'kafka.serde.class'='org.apache.hadoop.hive.serde2.JsonSerDe',
> 'kafka.topic'='hit-topic-1',
> 'kafka.write.semantic'='AT_LEAST_ONCE');
> SELECT COUNT(*) FROM kafka_table WHERE `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '10' MINUTES); # works due to
> HIVE-23408
> insert into kafka_table values(NULL, 'comment', 0, 1, 2, 3.0, NULL, NULL,
> NULL, NULL); # fails
> {code}
> exception I get:
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.kafkaesque.common.KafkaException: Failed to construct kafka
> producer
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:829)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1004)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 20 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.kafkaesque.common.KafkaException: Failed to construct kafka
> producer
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:872)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:823)
> ... 35 more
> Caused by: org.apache.kafkaesque.common.KafkaException: Failed to construct
>
[jira] [Updated] (HIVE-25553) Support Map data-type natively in Arrow format
[ https://issues.apache.org/jira/browse/HIVE-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-25553: Affects Version/s: 3.1.0 3.1.2 > Support Map data-type natively in Arrow format > -- > > Key: HIVE-25553 > URL: https://issues.apache.org/jira/browse/HIVE-25553 > Project: Hive > Issue Type: Improvement > Components: llap, Serializers/Deserializers >Affects Versions: 3.1.0, 3.1.2 >Reporter: Adesh Kumar Rao >Assignee: Sruthi Mooriyathvariam >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs > data-type (where stuct is containing the key-value pair of the map). This > causes issues when reading Map datatype using llap-ext-client as it reads a > list of structs instead. > HiveWarehouseConnector which uses the llap-ext-client throws exception when > the schema (containing Map data type) is different from actual data (list of > structs). > > Fixing this issue requires upgrading arrow version (where map data-type is > supported), modifying ArrowColumnarBatchSerDe and corresponding > Serializer/Deserializer to not use list as a workaround for map and use the > arrow map data-type instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25553) Support Map data-type natively in Arrow format
[ https://issues.apache.org/jira/browse/HIVE-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434716#comment-17434716 ] Sankar Hariappan commented on HIVE-25553: - Merged PR #2751 to master. Thanks [~warriersruthi]! > Support Map data-type natively in Arrow format > -- > > Key: HIVE-25553 > URL: https://issues.apache.org/jira/browse/HIVE-25553 > Project: Hive > Issue Type: Improvement > Components: llap, Serializers/Deserializers >Reporter: Adesh Kumar Rao >Assignee: Sruthi Mooriyathvariam >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs > data-type (where stuct is containing the key-value pair of the map). This > causes issues when reading Map datatype using llap-ext-client as it reads a > list of structs instead. > HiveWarehouseConnector which uses the llap-ext-client throws exception when > the schema (containing Map data type) is different from actual data (list of > structs). > > Fixing this issue requires upgrading arrow version (where map data-type is > supported), modifying ArrowColumnarBatchSerDe and corresponding > Serializer/Deserializer to not use list as a workaround for map and use the > arrow map data-type instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25637) Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
[
https://issues.apache.org/jira/browse/HIVE-25637?focusedWorklogId=670529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670529
]
ASF GitHub Bot logged work on HIVE-25637:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 07:55
Start Date: 27/Oct/21 07:55
Worklog Time Spent: 10m
Work Description: abstractdog opened a new pull request #2753:
URL: https://github.com/apache/hive/pull/2753
Change-Id: Ie6e83f196151a1ee64ad7cc32e4c7a6ee53fdb92
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670529)
Remaining Estimate: 0h
Time Spent: 10m
> Hive on Tez: inserting data failing into the non native hive external table
> managed by kafka storage handler
> -
>
> Key: HIVE-25637
> URL: https://issues.apache.org/jira/browse/HIVE-25637
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is the followup for HIVE-23408, repro is below:
> {code}
> CREATE EXTERNAL TABLE `kafka_table`(
> `timestamp` timestamp COMMENT 'from deserializer',
> `page` string COMMENT 'from deserializer',
> `newpage` boolean COMMENT 'from deserializer',
> `added` int COMMENT 'from deserializer',
> `deleted` bigint COMMENT 'from deserializer',
> `delta` double COMMENT 'from deserializer')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.kafka.KafkaSerDe'
> STORED BY
> 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> WITH SERDEPROPERTIES (
> 'serialization.format'='1')
> LOCATION
>
> 'hdfs://lbodorkafkaunsec-2.lbodorkafkaunsec.root.hwx.site:8020/warehouse/tablespace/external/hive/kafka_table'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'hive.kafka.max.retries'='6',
> 'hive.kafka.metadata.poll.timeout.ms'='3',
> 'hive.kafka.optimistic.commit'='false',
> 'hive.kafka.poll.timeout.ms'='5000',
>
> 'kafka.bootstrap.servers'='lbodorkafkaunsec-1.lbodorkafkaunsec.root.hwx.site:9092,lbodorkafkaunsec-2.lbodorkafkaunsec.root.hwx.site:9092,lbodorkafkaunsec-3.lbodorkafkaunsec.root.hwx.site:9092',
> 'kafka.serde.class'='org.apache.hadoop.hive.serde2.JsonSerDe',
> 'kafka.topic'='hit-topic-1',
> 'kafka.write.semantic'='AT_LEAST_ONCE');
> SELECT COUNT(*) FROM kafka_table WHERE `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '10' MINUTES); # works due to
> HIVE-23408
> insert into kafka_table values(NULL, 'comment', 0, 1, 2, 3.0, NULL, NULL,
> NULL, NULL); # fails
> {code}
> exception I get:
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.kafkaesque.common.KafkaException: Failed to construct kafka
> producer
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:829)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1004)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(Se
[jira] [Resolved] (HIVE-25553) Support Map data-type natively in Arrow format
[ https://issues.apache.org/jira/browse/HIVE-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-25553. - Resolution: Fixed > Support Map data-type natively in Arrow format > -- > > Key: HIVE-25553 > URL: https://issues.apache.org/jira/browse/HIVE-25553 > Project: Hive > Issue Type: Improvement > Components: llap, Serializers/Deserializers >Reporter: Adesh Kumar Rao >Assignee: Sruthi Mooriyathvariam >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs > data-type (where stuct is containing the key-value pair of the map). This > causes issues when reading Map datatype using llap-ext-client as it reads a > list of structs instead. > HiveWarehouseConnector which uses the llap-ext-client throws exception when > the schema (containing Map data type) is different from actual data (list of > structs). > > Fixing this issue requires upgrading arrow version (where map data-type is > supported), modifying ArrowColumnarBatchSerDe and corresponding > Serializer/Deserializer to not use list as a workaround for map and use the > arrow map data-type instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25553) Support Map data-type natively in Arrow format
[ https://issues.apache.org/jira/browse/HIVE-25553?focusedWorklogId=670527&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670527 ] ASF GitHub Bot logged work on HIVE-25553: - Author: ASF GitHub Bot Created on: 27/Oct/21 07:54 Start Date: 27/Oct/21 07:54 Worklog Time Spent: 10m Work Description: sankarh merged pull request #2751: URL: https://github.com/apache/hive/pull/2751 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 670527) Time Spent: 3h (was: 2h 50m) > Support Map data-type natively in Arrow format > -- > > Key: HIVE-25553 > URL: https://issues.apache.org/jira/browse/HIVE-25553 > Project: Hive > Issue Type: Improvement > Components: llap, Serializers/Deserializers >Reporter: Adesh Kumar Rao >Assignee: Sruthi Mooriyathvariam >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Currently ArrowColumnarBatchSerDe converts map datatype as a list of structs > data-type (where stuct is containing the key-value pair of the map). This > causes issues when reading Map datatype using llap-ext-client as it reads a > list of structs instead. > HiveWarehouseConnector which uses the llap-ext-client throws exception when > the schema (containing Map data type) is different from actual data (list of > structs). > > Fixing this issue requires upgrading arrow version (where map data-type is > supported), modifying ArrowColumnarBatchSerDe and corresponding > Serializer/Deserializer to not use list as a workaround for map and use the > arrow map data-type instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25578) Tests are failing because operators can't be closed
[ https://issues.apache.org/jira/browse/HIVE-25578?focusedWorklogId=670512&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670512 ] ASF GitHub Bot logged work on HIVE-25578: - Author: ASF GitHub Bot Created on: 27/Oct/21 07:25 Start Date: 27/Oct/21 07:25 Worklog Time Spent: 10m Work Description: kgyrtkirk closed pull request #2688: URL: https://github.com/apache/hive/pull/2688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 670512) Time Spent: 20m (was: 10m) > Tests are failing because operators can't be closed > --- > > Key: HIVE-25578 > URL: https://issues.apache.org/jira/browse/HIVE-25578 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The following qtests are failing consistently > ([example|http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2667/6/tests/]) > on the master branch: > * TestMiniLlapCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/420/]) > ** newline > ** groupby_bigdata > ** input20 > ** input33 > ** rcfile_bigdata > ** remote_script > * TestContribCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/421/]) > ** serde_typedbytes5 > The failure reason for all seems to be that operators can't be closed. Not > 100% sure that TestContribCliDriver#serde_typedbytes5 failure is related to > the others – the issue seems to be the same, the error message is a bit > different. > I'm about to disable these as they are blocking all work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25633) Prevent shutdown of MetaStore scheduled worker ThreadPool
[
https://issues.apache.org/jira/browse/HIVE-25633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-25633.
-
Fix Version/s: 4.0.0
Resolution: Fixed
merged into master. Thank you [~euigeun_chung] and [~kkasa] for reviewing the
changes!
> Prevent shutdown of MetaStore scheduled worker ThreadPool
> -
>
> Key: HIVE-25633
> URL: https://issues.apache.org/jira/browse/HIVE-25633
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> [~lpinter] have noticed that this patch has some sideffect:
> in HIVE-23164 the patch have added a {{ThreadPool#shutdown}} to
> {{HMSHandler#shutdown}} - which could cause trouble in case a {{HMSHandler}}
> is shutdown and a new one is created
> I was looking for cases in which a HMSHandler is created inside the metastore
> (beyond the one HiveMetaStore is using) - and I think tasks like
> PartitionManagementTask which uses Msck which uses HMC to access the
> metastore - and they close the client - which closes the hmshandler client ;
> which will shut down the threadpool
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25633) Prevent shutdown of MetaStore scheduled worker ThreadPool
[
https://issues.apache.org/jira/browse/HIVE-25633?focusedWorklogId=670508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670508
]
ASF GitHub Bot logged work on HIVE-25633:
-
Author: ASF GitHub Bot
Created on: 27/Oct/21 07:19
Start Date: 27/Oct/21 07:19
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2737:
URL: https://github.com/apache/hive/pull/2737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 670508)
Time Spent: 0.5h (was: 20m)
> Prevent shutdown of MetaStore scheduled worker ThreadPool
> -
>
> Key: HIVE-25633
> URL: https://issues.apache.org/jira/browse/HIVE-25633
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> [~lpinter] have noticed that this patch has some sideffect:
> in HIVE-23164 the patch have added a {{ThreadPool#shutdown}} to
> {{HMSHandler#shutdown}} - which could cause trouble in case a {{HMSHandler}}
> is shutdown and a new one is created
> I was looking for cases in which a HMSHandler is created inside the metastore
> (beyond the one HiveMetaStore is using) - and I think tasks like
> PartitionManagementTask which uses Msck which uses HMC to access the
> metastore - and they close the client - which closes the hmshandler client ;
> which will shut down the threadpool
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
