[jira] [Created] (HIVE-17389) Yetus is always failing on rat checks
Barna Zsombor Klara created HIVE-17389: -- Summary: Yetus is always failing on rat checks Key: HIVE-17389 URL: https://issues.apache.org/jira/browse/HIVE-17389 Project: Hive Issue Type: Bug Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Trivial Rat checks are failing on metastore_db/dblock and files under patchprocess created by Yetus itself. Both directories should be excluded from rat checks. CC: [~pvary] [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17382) Change startsWith relation introduced in HIVE-17316
Barna Zsombor Klara created HIVE-17382: -- Summary: Change startsWith relation introduced in HIVE-17316 Key: HIVE-17382 URL: https://issues.apache.org/jira/browse/HIVE-17382 Project: Hive Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Fix For: 3.0.0 In HiveConf the new name should be checked if it starts with a restricted/hidden variable prefix and not vice-versa. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17356) Missing ASF headers 3 classes
Barna Zsombor Klara created HIVE-17356: -- Summary: Missing ASF headers 3 classes Key: HIVE-17356 URL: https://issues.apache.org/jira/browse/HIVE-17356 Project: Hive Issue Type: Bug Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Trivial JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java are missing the ASF header that should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17337) Investigate the source of the flakiness of BeeLineTests when executed in parallel
Barna Zsombor Klara created HIVE-17337: -- Summary: Investigate the source of the flakiness of BeeLineTests when executed in parallel Key: HIVE-17337 URL: https://issues.apache.org/jira/browse/HIVE-17337 Project: Hive Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara BeeLine qtests occasionally fail with unexplained diffs in the output when run in parallel. HIVE-17322 removed the parallelisation, but we should investigate the root cause of the issue before we create many more BeeLine tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17322) Execute BeeLine qtests in a serial manner to prevent flakyness
Barna Zsombor Klara created HIVE-17322: -- Summary: Execute BeeLine qtests in a serial manner to prevent flakyness Key: HIVE-17322 URL: https://issues.apache.org/jira/browse/HIVE-17322 Project: Hive Issue Type: Bug Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
Barna Zsombor Klara created HIVE-17319: -- Summary: Make BoneCp configurable using hive properties in hive-site.xml Key: HIVE-17319 URL: https://issues.apache.org/jira/browse/HIVE-17319 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17318) Make Hikari configurable using hive properties in hive-site.xml
Barna Zsombor Klara created HIVE-17318: -- Summary: Make Hikari configurable using hive properties in hive-site.xml Key: HIVE-17318 URL: https://issues.apache.org/jira/browse/HIVE-17318 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
Barna Zsombor Klara created HIVE-17317: -- Summary: Make Dbcp configurable using hive properties in hive-site.xml Key: HIVE-17317 URL: https://issues.apache.org/jira/browse/HIVE-17317 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17316) Use regular expressions for the hidden configuration variables
Barna Zsombor Klara created HIVE-17316: -- Summary: Use regular expressions for the hidden configuration variables Key: HIVE-17316 URL: https://issues.apache.org/jira/browse/HIVE-17316 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Currently HiveConf variables which should not be displayed to the user need to be enumerated. We should enhance this to be able to set regular expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17315) Make the DataSource used by the DataNucleus in the HMS configurable using Hive properties
Barna Zsombor Klara created HIVE-17315: -- Summary: Make the DataSource used by the DataNucleus in the HMS configurable using Hive properties Key: HIVE-17315 URL: https://issues.apache.org/jira/browse/HIVE-17315 Project: Hive Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Currently we may use several connection pool implementations in the backend (hikari, dbCp, boneCp) but these can only be configured using proprietary xml files and not through hive-site.xml like DataNucleus. We should make them configurable just like DataNucleus, by allowing Hive properties prefix by hikari, dbcp, bonecp to be set in the hive-site.xml. However since these configurations may contain sensitive information (passwords) these properties should not be displayable or manually settable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
Barna Zsombor Klara created HIVE-17305: -- Summary: New insert overwrite dynamic partitions qtest need to have the golden file regenerated Key: HIVE-17305 URL: https://issues.apache.org/jira/browse/HIVE-17305 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Trivial -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17267) Make HMS Notification Listeners typesafe
Barna Zsombor Klara created HIVE-17267: -- Summary: Make HMS Notification Listeners typesafe Key: HIVE-17267 URL: https://issues.apache.org/jira/browse/HIVE-17267 Project: Hive Issue Type: Bug Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Currently in the HMS we support two types of notification listeners, transactional and non-transactional ones. Transactional listeners will only be invoked if the jdbc transaction finished successfully while non-transactional ones are supposed to be resilient and will be invoked in any case, even for failures. Having the same type for these two is a source of confusion and opens the door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17263) Reduce debug logging for S3 tables
Barna Zsombor Klara created HIVE-17263: -- Summary: Reduce debug logging for S3 tables Key: HIVE-17263 URL: https://issues.apache.org/jira/browse/HIVE-17263 Project: Hive Issue Type: Improvement Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Minor When log level is set to debug operations accessing tables on amazon s3 will output a significant amount of logs, a lot of which is about the http communication (http headers and requests) which may not be that useful even for debugging purposes. Since some ZooKeeper, Hadoop, DataNucleus etc. loggers are by default set to INFO+ levels I suggest we do the same for Apache Http and AWS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17262) Use Hdfs API instead of directory listing to check snapshots for a path
Barna Zsombor Klara created HIVE-17262: -- Summary: Use Hdfs API instead of directory listing to check snapshots for a path Key: HIVE-17262 URL: https://issues.apache.org/jira/browse/HIVE-17262 Project: Hive Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara HIVE-11667 introduced a check for snapshots of a Hdfs directory to enable truncating snapshotted Hive tables. https://issues.apache.org/jira/browse/HDFS-12257 will introduce a public API to check for the same. Once it is resolved we should switch to it in Hive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17149) Hdfs directory is not cleared if partition creation failed on HMS
Barna Zsombor Klara created HIVE-17149: -- Summary: Hdfs directory is not cleared if partition creation failed on HMS Key: HIVE-17149 URL: https://issues.apache.org/jira/browse/HIVE-17149 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Hive#loadPartition will load a directory into a Hive Table Partition. It will alter the existing content of the partition with the new contents and create a new partition if one does not exist. The file move is performed before the partition creation and if the creation failes, the moved files are not cleared. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17052) Remove logging of predicate filters
Barna Zsombor Klara created HIVE-17052: -- Summary: Remove logging of predicate filters Key: HIVE-17052 URL: https://issues.apache.org/jira/browse/HIVE-17052 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 3.0.0 Reporter: Barna Zsombor Klara HIVE-16869 added the filter predicate to the debug log of HS2, but since these filters may contain sensitive information they should not be logged out. The log statement should be changed back to the original form. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17001) Insert overwrite table doesn't clean partition directory on HDFS if partition is missing from HMS
Barna Zsombor Klara created HIVE-17001:
--
Summary: Insert overwrite table doesn't clean partition directory
on HDFS if partition is missing from HMS
Key: HIVE-17001
URL: https://issues.apache.org/jira/browse/HIVE-17001
Project: Hive
Issue Type: Bug
Components: HiveServer2, Metastore
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Insert overwrite table should clear existing data before creating the new data
files.
For a partitioned table we will clean any folder of existing partitions on
HDFS, however if the partition folder exists only on HDFS and the partition
definition is missing in HMS, the folder is not cleared.
Reproduction steps:
1. CREATE TABLE test( col1 string) PARTITIONED BY (ds string);
2. INSERT INTO test PARTITION(ds='p1') values ('a');
3. Copy the data to a different folder with different name.
4. ALTER TABLE test DROP PARTITION (ds='p1');
5. Recreate the partition directory, copy and rename the data file back
6. INSERT INTO test PARTITION(ds='p1') values ('b');
7. SELECT * from test;
will result in 2 records being returned instead of 1.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-16959) Flaky Test : TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
Barna Zsombor Klara created HIVE-16959: -- Summary: Flaky Test : TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] Key: HIVE-16959 URL: https://issues.apache.org/jira/browse/HIVE-16959 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Barna Zsombor Klara Test failed on the pre-commit but runs locally. Error Message Client result comparison failed with error code = 1 while executing fname=insert_overwrite_local_directory_1 1172d1171 < k21=v21#k22=v22#k31=v31:foo2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-16931) Flaky test: TestPerfCliDriver.query14, query16, query94
Barna Zsombor Klara created HIVE-16931:
--
Summary: Flaky test: TestPerfCliDriver.query14, query16, query94
Key: HIVE-16931
URL: https://issues.apache.org/jira/browse/HIVE-16931
Project: Hive
Issue Type: Sub-task
Reporter: Barna Zsombor Klara
Diffs are similar for all 3 tests:
{code}
1c1
< Warning: Shuffle Join MERGEJOIN[107][tables = [$hdt$_2, $hdt$_3, $hdt$_1,
$hdt$_4]] in Stage 'Reducer 17' is a cross product
---
> Warning: Shuffle Join MERGEJOIN[113][tables = [$hdt$_2, $hdt$_3, $hdt$_1,
> $hdt$_4]] in Stage 'Reducer 18' is a cross product
65,74c65,74
< Reducer 13 <- Map 12 (SIMPLE_EDGE)
< Reducer 15 <- Map 14 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
< Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
< Reducer 17 <- Map 14 (CUSTOM_SIMPLE_EDGE), Map 19 (CUSTOM_SIMPLE_EDGE), Map
20 (CUSTOM_SIMPLE_EDGE), Map 21 (CUSTOM_SIMPLE_EDGE)
< Reducer 18 <- Reducer 17 (SIMPLE_EDGE)
< Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
< Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
< Reducer 4 <- Map 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
< Reducer 5 <- Reducer 13 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
< Reducer 6 <- Reducer 16 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
---
> Reducer 14 <- Map 13 (SIMPLE_EDGE)
> Reducer 16 <- Map 15 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
> Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
> Reducer 18 <- Map 15 (CUSTOM_SIMPLE_EDGE), Map 20 (CUSTOM_SIMPLE_EDGE), Map
> 21 (CUSTOM_SIMPLE_EDGE), Map 22 (CUSTOM_SIMPLE_EDGE)
> Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
> Reducer 3 <- Map 11 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
> Reducer 4 <- Map 12 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 14 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 17 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
76c76,77
< Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
---
> Reducer 8 <- Reducer 7 (CUSTOM_SIMPLE_EDGE)
> Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
82c83
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-16718) Provide a way to pass in user supplied maven build and test arguments to Ptest
Barna Zsombor Klara created HIVE-16718: -- Summary: Provide a way to pass in user supplied maven build and test arguments to Ptest Key: HIVE-16718 URL: https://issues.apache.org/jira/browse/HIVE-16718 Project: Hive Issue Type: New Feature Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Minor Fix For: 3.0.0 Currently we can only pass in maven build and test arguments from the properties file, so all of them need to be hardcoded. We should find a way to pass in arguments from the command line. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16679) Missing ASF header on properties file in ptest2 project
Barna Zsombor Klara created HIVE-16679:
--
Summary: Missing ASF header on properties file in ptest2 project
Key: HIVE-16679
URL: https://issues.apache.org/jira/browse/HIVE-16679
Project: Hive
Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial
Fix For: 3.0.0
The ASF header is missing on
{{testutils/ptest2//conf/deployed/master-mr2.properties}} causing the build of
the ptest2 project to fail on a RAT check.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HIVE-16559) Parquet schema evolution for partitioned tables may break if table and partition serdes differ
Barna Zsombor Klara created HIVE-16559:
--
Summary: Parquet schema evolution for partitioned tables may break
if table and partition serdes differ
Key: HIVE-16559
URL: https://issues.apache.org/jira/browse/HIVE-16559
Project: Hive
Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Parquet schema evolution should make it possible to have partitions/tables
backed by files with different schemas. Hive should match the table columns
with file columns based on the column name if possible.
However if the serde for a table is missing columns from the serde of a
partition Hive fails to match the columns together.
Steps to reproduce:
{code}
CREATE TABLE myparquettable_parted
(
name string,
favnumber int,
favcolor string,
age int,
favpet string
)
PARTITIONED BY (day string)
STORED AS PARQUET;
INSERT OVERWRITE TABLE myparquettable_parted
PARTITION(day='2017-04-04')
SELECT
'mary' as name,
5 AS favnumber,
'blue' AS favcolor,
35 AS age,
'dog' AS favpet;
REPLACE COLUMNS
(
favnumber int,
age int
);
[jira] [Created] (HIVE-16469) Parquet timestamp table property is not always taken into account
Barna Zsombor Klara created HIVE-16469: -- Summary: Parquet timestamp table property is not always taken into account Key: HIVE-16469 URL: https://issues.apache.org/jira/browse/HIVE-16469 Project: Hive Issue Type: Bug Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara The parquet timestamp timezone property is currently copied over into the JobConf in the FetchOperator, but this may be too late for some execution paths. We should: 1 - copy the property over earlier 2 - set the default value on the JobConf if no property is set, and fail in the ParquetRecordReader if the property is missing from the JobConf -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16357) Failed folder creation when creating a new table is reported incorrectly
Barna Zsombor Klara created HIVE-16357:
--
Summary: Failed folder creation when creating a new table is
reported incorrectly
Key: HIVE-16357
URL: https://issues.apache.org/jira/browse/HIVE-16357
Project: Hive
Issue Type: Bug
Reporter: Barna Zsombor Klara
If the directory for a Hive table could not be created, them the HMS will throw
a metaexception:
{code}
if (tblPath != null) {
if (!wh.isDir(tblPath)) {
if (!wh.mkdirs(tblPath, true)) {
throw new MetaException(tblPath
+ " is not a directory or unable to create one");
}
madeDir = true;
}
}
{code}
However in the finally block we always try to call the DbNotificationListener,
which in turn will also throw an exception because the directory is missing,
overwriting the initial exception with a FileNotFoundException.
Actual stacktrace seen by the caller:
{code}
2017-04-03T05:58:00,128 ERROR [pool-7-thread-2] metastore.RetryingHMSHandler:
MetaException(message:java.lang.RuntimeException:
java.io.FileNotFoundException: File file:/.../0 does not exist)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6074)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1496)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy28.create_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11125)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11109)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
file:/.../0 does not exist
at
org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.(DbNotificationListener.java:203)
at
org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:137)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1463)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1482)
... 20 more
Caused by: java.io.FileNotFoundException: File file:/.../0 does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:574)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:243)
at
org.apache.hadoop.fs.ProxyFileSystem.listStatus(ProxyFileSystem.java:195)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:243)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
at
org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.(DbNotificationListener.java:200)
... 23 more
{code}
The original exception should be thrown back to the caller while any
[jira] [Created] (HIVE-16312) Flaky test: TestHCatClient.testTransportFailure
Barna Zsombor Klara created HIVE-16312:
--
Summary: Flaky test: TestHCatClient.testTransportFailure
Key: HIVE-16312
URL: https://issues.apache.org/jira/browse/HIVE-16312
Project: Hive
Issue Type: Sub-task
Reporter: Barna Zsombor Klara
The test has been failing consistently for 10+ builds.
Error message:
{code}
Error Message
The expected exception was never thrown.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HIVE-16256) Flaky test: TestCliDriver.testCliDriver[comments]
Barna Zsombor Klara created HIVE-16256:
--
Summary: Flaky test: TestCliDriver.testCliDriver[comments]
Key: HIVE-16256
URL: https://issues.apache.org/jira/browse/HIVE-16256
Project: Hive
Issue Type: Sub-task
Reporter: Barna Zsombor Klara
Test has been failing for 6 consecutive runs. Most recent:
https://builds.apache.org/job/PreCommit-HIVE-Build/4245/testReport/
Diff:
147a148
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
167a169
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HIVE-16231) Parquet timestamp may be stored differently since HIVE-12767
Barna Zsombor Klara created HIVE-16231: -- Summary: Parquet timestamp may be stored differently since HIVE-12767 Key: HIVE-16231 URL: https://issues.apache.org/jira/browse/HIVE-16231 Project: Hive Issue Type: Bug Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Critical If the parquet table is missing its timezone property then the timestamp will be stored with an adjustment instead of without it. This will cause a regression with other applications like Impala or Spark. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16088) Fix hive conf property name introduced in HIVE-12767
Barna Zsombor Klara created HIVE-16088:
--
Summary: Fix hive conf property name introduced in HIVE-12767
Key: HIVE-16088
URL: https://issues.apache.org/jira/browse/HIVE-16088
Project: Hive
Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial
The configuration property {{parquet.mr.int96.enable.utc.write.zone}} should be
called {{hive.parquet.mr.int96.enable.utc.write.zone}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HIVE-16024) MSCK Repair Requires nonstrict hive.mapred.mode
Barna Zsombor Klara created HIVE-16024: -- Summary: MSCK Repair Requires nonstrict hive.mapred.mode Key: HIVE-16024 URL: https://issues.apache.org/jira/browse/HIVE-16024 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 2.2.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara MSCK repair fails when hive.mapred.mode is set to strict HIVE-13788 modified the way we read up partitions for a table to improve performance. Unfortunately it is using PartitionPruner to load the partitions which in turn is checking hive.mapred.mode. The previous code did not check hive.mapred.mode. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15864) Fix type introduced in HIVE-14754
Barna Zsombor Klara created HIVE-15864: -- Summary: Fix type introduced in HIVE-14754 Key: HIVE-15864 URL: https://issues.apache.org/jira/browse/HIVE-15864 Project: Hive Issue Type: Sub-task Affects Versions: 2.2.0 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara hs2_suceeded_queries needs another "c": hs2_succeeded_queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15616) Improve contents of qfile test output
Barna Zsombor Klara created HIVE-15616: -- Summary: Improve contents of qfile test output Key: HIVE-15616 URL: https://issues.apache.org/jira/browse/HIVE-15616 Project: Hive Issue Type: Improvement Components: Tests Affects Versions: 2.1.1 Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Priority: Minor The current output of the failed qtests has a less than ideal signal to noise ratio. We have duplicated stack traces and messages between the error message/stack trace/error out. For diff errors the actual difference is missing from the error message and can be found only in the standard out. I would like to simplify this output by removing duplications, moving relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15387) NPE in HiveServer2 webUI Historical SQL Operations section
Barna Zsombor Klara created HIVE-15387:
--
Summary: NPE in HiveServer2 webUI Historical SQL Operations section
Key: HIVE-15387
URL: https://issues.apache.org/jira/browse/HIVE-15387
Project: Hive
Issue Type: Bug
Reporter: Barna Zsombor Klara
Priority: Minor
The runtime value on a SQLOperationDisplay may be null, which may lead to NPEs
on the web UI.
Stack trace:
{code}
java.lang.NullPointerException
at
org.apache.hive.generated.hiveserver2.hiveserver2_jsp._jspService(hiveserver2_jsp.java:145)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:745)
{code}
Compiled jsp segment:
{code}
124out.print(
conf.get(ConfVars.HIVE_SERVER2_WEBUI_MAX_HISTORIC_QUERIES.varname) );
125out.write(" Closed Queries\n\n\nUser Name\n
Query\nExecution Engine\nState\n
Opened (s)\nClosed Timestamp\nLatency
(s)\nDrilldown Link\n\n");
126
127queries = 0;
128operations =
sessionManager.getOperationManager().getHistoricalSQLOperations();
129for (SQLOperationDisplay operation : operations) {
130queries++;
131
132out.write("\n\n");
133out.print( operation.getUserName() );
134out.write("\n");
135out.print( operation.getQueryDisplay() == null ? "Unknown" :
operation.getQueryDisplay().getQueryString() );
136out.write("\n");
137out.print( operation.getExecutionEngine() );
138out.write("\n");
139out.print( operation.getState() );
140out.write("\n");
141out.print( operation.getElapsedTime()/1000 );
142out.write("\n");
143out.print( operation.getEndTime() == null ? "In Progress" : new
Date(operation.getEndTime()) );
144out.write("\n");
145out.print( operation.getRuntime()/1000 );
146out.write("\n");
{code}
Still trying to find a way to easily reproduce the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15170) Flaky test: TestEmbeddedThriftBinaryCLIService.testTaskStatus
Barna Zsombor Klara created HIVE-15170:
--
Summary: Flaky test:
TestEmbeddedThriftBinaryCLIService.testTaskStatus
Key: HIVE-15170
URL: https://issues.apache.org/jira/browse/HIVE-15170
Project: Hive
Issue Type: Sub-task
Reporter: Barna Zsombor Klara
Test failed in:
https://builds.apache.org/job/PreCommit-HIVE-Build/2043/
https://issues.apache.org/jira/browse/HIVE-12891 (Test results have been
deleted but the test failed in two subsequent runs without a change to the
patch file)
Logs:
{code}
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: PREHOOK: query: CREATE TABLE TEST_EXEC_ASYNC(ID STRING)
PREHOOK: type: CREATETABLE
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: PREHOOK: Output: database:default
PREHOOK: Output: default@TEST_EXEC_ASYNC
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: PREHOOK: Output: default@TEST_EXEC_ASYNC
2016-11-08T18:39:31,369 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: POSTHOOK: query: CREATE TABLE TEST_EXEC_ASYNC(ID STRING)
POSTHOOK: type: CREATETABLE
2016-11-08T18:39:31,370 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
2016-11-08T18:39:31,370 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: POSTHOOK: Output: database:default
POSTHOOK: Output: default@TEST_EXEC_ASYNC
2016-11-08T18:39:31,370 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main]
SessionState: POSTHOOK: Output: default@TEST_EXEC_ASYNC
...
2016-11-08T18:39:31,711 ERROR [HiveServer2-Background-Pool: Thread-4824]
SessionState: PREHOOK: query: select a.id, b.id from (SELECT ID + ' ' `ID` FROM
TEST_EXEC_ASYNC) a full outer join (SELECT ID + ' ' `ID` FROM TEST_EXEC_ASYNC)
b on a.ID=b.ID
PREHOOK: type: QUERY
2016-11-08T18:39:31,711 ERROR [HiveServer2-Background-Pool: Thread-4824]
SessionState: PREHOOK: type: QUERY
PREHOOK: Input: default@test_exec_async
2016-11-08T18:39:31,711 ERROR [HiveServer2-Background-Pool: Thread-4824]
SessionState: PREHOOK: Input: default@test_exec_async
PREHOOK: Output:
file:/home/hiveptest/54.177.123.16-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/167769e0-ccac-41b0-b271-4175e0c37a79/hive_2016-11-08_18-39-31_373_2208326672728744152-1/-mr-10001
2016-11-08T18:39:31,712 ERROR [HiveServer2-Background-Pool: Thread-4824]
SessionState: PREHOOK: Output:
file:/home/hiveptest/54.177.123.16-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/167769e0-ccac-41b0-b271-4175e0c37a79/hive_2016-11-08_18-39-31_373_2208326672728744152-1/-mr-10001
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15169) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver.columnstats_part_coltype
Barna Zsombor Klara created HIVE-15169: -- Summary: Flaky test: TestMiniLlapLocalCliDriver.testCliDriver.columnstats_part_coltype Key: HIVE-15169 URL: https://issues.apache.org/jira/browse/HIVE-15169 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara Flaky runs: https://builds.apache.org/job/PreCommit-HIVE-Build/2043/ https://issues.apache.org/jira/browse/HIVE-12891 (Test report missing, but test failed in a second run without a new patch file being uploaded) Error Message Unexpected exception java.lang.AssertionError: Client execution failed with error code = 4 running analyze table partcolstatsnum partition (tint=100, sint=1000, bint=100) compute statistics for columnsfname=columnstats_part_coltype.q -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)
Barna Zsombor Klara created HIVE-15168:
--
Summary: Flaky test: TestSparkClient.testJobSubmission (still
flaky)
Key: HIVE-15168
URL: https://issues.apache.org/jira/browse/HIVE-15168
Project: Hive
Issue Type: Sub-task
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
[HIVE-14910|https://issues.apache.org/jira/browse/HIVE-14910] already addressed
one source of flakyness bud sadly not all it seems.
In JobHandleImpl the listeners are registered after the job has been submitted.
This may end up in a racecondition.
{code}
// Link the RPC and the promise so that events from one are propagated to the
other as
// needed.
rpc.addListener(new
GenericFutureListener() {
@Override
public void operationComplete(io.netty.util.concurrent.Future f) {
if (f.isSuccess()) {
handle.changeState(JobHandle.State.QUEUED);
} else if (!promise.isDone()) {
promise.setFailure(f.cause());
}
}
});
promise.addListener(new GenericFutureListener() {
@Override
public void operationComplete(Promise p) {
if (jobId != null) {
jobs.remove(jobId);
}
if (p.isCancelled() && !rpc.isDone()) {
rpc.cancel(true);
}
}
});
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15116) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver.join_acid_non_acid
Barna Zsombor Klara created HIVE-15116:
--
Summary: Flaky test:
TestMiniLlapLocalCliDriver.testCliDriver.join_acid_non_acid
Key: HIVE-15116
URL: https://issues.apache.org/jira/browse/HIVE-15116
Project: Hive
Issue Type: Sub-task
Reporter: Barna Zsombor Klara
{code}
Running: diff -a
/home/hiveptest/54.193.134.5-hiveptest-0/apache-github-source-source/itests/qtest/target/qfile-results/clientpositive/join_acid_non_acid.q.out
/home/hiveptest/54.193.134.5-hiveptest-0/apache-github-source-source/ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out
73d72
< 1 a
74a74
> 1 a
{code}
Seems to be a white space difference.
The test failed in the following pre-commit runs:
https://builds.apache.org/job/PreCommit-HIVE-Build/1932/testReport/
https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport/
https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15115) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
Barna Zsombor Klara created HIVE-15115: -- Summary: Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] Key: HIVE-15115 URL: https://issues.apache.org/jira/browse/HIVE-15115 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara This test was identified as flaky before, it seems it turned flaky again. Earlier Jira: [HIVE-14976|https://issues.apache.org/jira/browse/HIVE-14976] New flaky runs: https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14775) Investigate IOException usage in Metrics APIs
Barna Zsombor Klara created HIVE-14775: -- Summary: Investigate IOException usage in Metrics APIs Key: HIVE-14775 URL: https://issues.apache.org/jira/browse/HIVE-14775 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara A large number of metrics APIs seems to declare to throw IOExceptions needlessly. (incrementCounter, decrementCounter etc.) This is not only misleading but it fills up the code with unnecessary catch blocks never to be reached. We should investigate if these exceptions are thrown at all, and remove them if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14758) Metrics for threads/threading
Barna Zsombor Klara created HIVE-14758: -- Summary: Metrics for threads/threading Key: HIVE-14758 URL: https://issues.apache.org/jira/browse/HIVE-14758 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara Track the nr. of thrift/http/executor threads at a given time in HS2. Track the nr. of completed and terminated threads for a time interval. Average thread lifetime. (Not sure if this is easily implementable). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14757) Metrics for Zookeeper locks/locking
Barna Zsombor Klara created HIVE-14757: -- Summary: Metrics for Zookeeper locks/locking Key: HIVE-14757 URL: https://issues.apache.org/jira/browse/HIVE-14757 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara Track the nr. of locks (shared or exclusive). Track the average time to acquire a lock. Track the number of queries waiting to acquire locks. Track the average time a lock exists once acquired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14756) Metrics for the HS2/HMS interface
Barna Zsombor Klara created HIVE-14756: -- Summary: Metrics for the HS2/HMS interface Key: HIVE-14756 URL: https://issues.apache.org/jira/browse/HIVE-14756 Project: Hive Issue Type: Sub-task Environment: We should be able to track the nr. of API calls for a given time period, how many of these were successful, retried or failed. Reporter: Barna Zsombor Klara -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14754) Track the queries execution lifecycle times
Barna Zsombor Klara created HIVE-14754: -- Summary: Track the queries execution lifecycle times Key: HIVE-14754 URL: https://issues.apache.org/jira/browse/HIVE-14754 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara We should be able to track the nr. of queries being compiled/executed at any given time, as well as the duration of the execution and compilation phase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14755) Track the success/failure rate of queries
Barna Zsombor Klara created HIVE-14755: -- Summary: Track the success/failure rate of queries Key: HIVE-14755 URL: https://issues.apache.org/jira/browse/HIVE-14755 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara Track the number of failed/successfully completed queries for a given time period. Time period should be configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14753) Track the number of open/closed/abandoned sessions in HS2
Barna Zsombor Klara created HIVE-14753: -- Summary: Track the number of open/closed/abandoned sessions in HS2 Key: HIVE-14753 URL: https://issues.apache.org/jira/browse/HIVE-14753 Project: Hive Issue Type: Sub-task Reporter: Barna Zsombor Klara We should be able to track the nr. of sessions since the startup of the HS2 instance as well as the average lifetime of a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14752) New metrics should be introduced to more easily monitor the workload of HS2/HMS
Barna Zsombor Klara created HIVE-14752: -- Summary: New metrics should be introduced to more easily monitor the workload of HS2/HMS Key: HIVE-14752 URL: https://issues.apache.org/jira/browse/HIVE-14752 Project: Hive Issue Type: Task Components: Hive, HiveServer2, Metastore Reporter: Barna Zsombor Klara Assignee: Barna Zsombor Klara Currently we have no way to easily track the number of sessions/queries in a HS2 instance. It would be great if we could tell the number of queries being run, which lifecycle phase they are in (submitted, compilation, execution), and how much time was spent executing/compiling them. This Jira will be an umbrella to track the subtasks for each potentially useful metric we can think of. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14601) Altering table/partition file format with preexisting data should not be allowed
Barna Zsombor Klara created HIVE-14601:
--
Summary: Altering table/partition file format with preexisting
data should not be allowed
Key: HIVE-14601
URL: https://issues.apache.org/jira/browse/HIVE-14601
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Minor
The file format of a table or a partition can be changed using an alter
statement. However this only affects the metadata, the data in hdfs is not
changed, leading to a table from which you cannot select anymore.
Changing the file format back fixes the issue, but a better approach would be
to prevent the alter to the file format if we have data in the tables.
The issue is reproducible by executing the following commands:
{code}
create table test (id int);
insert into test values (1);
alter table test set fileformat parquet;
insert into test values (2);
select * from test;
{code}
Will result in:
{code}
java.lang.RuntimeException: .../00_0 is not a Parquet file (too small)
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-14226) Invalid check on an ASTNode#toStringTree in CalcitePlanner
Barna Zsombor Klara created HIVE-14226: -- Summary: Invalid check on an ASTNode#toStringTree in CalcitePlanner Key: HIVE-14226 URL: https://issues.apache.org/jira/browse/HIVE-14226 Project: Hive Issue Type: Bug Reporter: Barna Zsombor Klara Priority: Minor The patch for HIVE-12590 has modified the string tree representation of an ASTNode, so that everything except string literals are now lowercase. However CalcitePlanner#genSelectLogicalPlan is still checking the occurrence of an uppercase TOK_FUNCTIONDI constant in the string tree. (Line 2997 in CalcitePlanner on master) I think this check will never evaluate to true, so we should either be using a lowercase String, or remove the check if it is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
