[jira] [Commented] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[
https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117928#comment-16117928
]
Hive QA commented on HIVE-17235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880774/HIVE-17235.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10995 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.serde2.io.TestHiveDecimalWritableVersion.testVerifyHiveDecimalWritablePublicMethodsAndFieldsVersions
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6290/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880774 - PreCommit-HIVE-Build
> Add ORC Decimal64 Serialization/Deserialization (Part 1)
>
>
> Key: HIVE-17235
> URL: https://issues.apache.org/jira/browse/HIVE-17235
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch,
> HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch
>
>
> The storage-api changes for ORC-209.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions
[ https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117910#comment-16117910 ] Lefty Leverenz commented on HIVE-16895: --- Doc note: This adds *hive.repl.partitions.dump.parallelism* to HiveConf.java, so it needs to be documented in the wiki. * [Configuration Properties -- Replication | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Replication] Added a TODOC3.0 label. > Multi-threaded execution of bootstrap dump of partitions > - > > Key: HIVE-16895 > URL: https://issues.apache.org/jira/browse/HIVE-16895 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16895.1.patch, HIVE-16895.2.patch > > > to allow faster execution of bootstrap dump phase we dump multiple partitions > from same table simultaneously. > even though dumping functions is not going to be a blocker, moving to > similar execution modes for all metastore objects will make code more > coherent. > Bootstrap dump at db level does : > * boostrap of all tables > ** boostrap of all partitions in a table. (scope of current jira) > * boostrap of all functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions
[ https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-16895: -- Labels: TODOC3.0 (was: ) > Multi-threaded execution of bootstrap dump of partitions > - > > Key: HIVE-16895 > URL: https://issues.apache.org/jira/browse/HIVE-16895 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16895.1.patch, HIVE-16895.2.patch > > > to allow faster execution of bootstrap dump phase we dump multiple partitions > from same table simultaneously. > even though dumping functions is not going to be a blocker, moving to > similar execution modes for all metastore objects will make code more > coherent. > Bootstrap dump at db level does : > * boostrap of all tables > ** boostrap of all partitions in a table. (scope of current jira) > * boostrap of all functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16294) Support snapshot for truncate table
[ https://issues.apache.org/jira/browse/HIVE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117892#comment-16117892 ] Lefty Leverenz commented on HIVE-16294: --- Should this be documented in the wiki? > Support snapshot for truncate table > --- > > Key: HIVE-16294 > URL: https://issues.apache.org/jira/browse/HIVE-16294 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Vihang Karajgaonkar >Assignee: Barna Zsombor Klara > Fix For: 3.0.0 > > Attachments: HIVE-16294.01.patch, HIVE-16294.02.patch, > HIVE-16294.03.patch, HIVE-16294.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17148) Incorrect result for Hive join query with COALESCE in WHERE condition
[
https://issues.apache.org/jira/browse/HIVE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117886#comment-16117886
]
Ashutosh Chauhan commented on HIVE-17148:
-
+1
> Incorrect result for Hive join query with COALESCE in WHERE condition
> -
>
> Key: HIVE-17148
> URL: https://issues.apache.org/jira/browse/HIVE-17148
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.1
>Reporter: Vlad Gudikov
>Assignee: Vlad Gudikov
> Attachments: HIVE-17148.1.patch, HIVE-17148.patch
>
>
> The issue exists in Hive-2.1. In Hive-1.2 the query works fine with cbo
> enabled:
> STEPS TO REPRODUCE:
> {code}
> Step 1: Create a table ct1
> create table ct1 (a1 string,b1 string);
> Step 2: Create a table ct2
> create table ct2 (a2 string);
> Step 3 : Insert following data into table ct1
> insert into table ct1 (a1) values ('1');
> Step 4 : Insert following data into table ct2
> insert into table ct2 (a2) values ('1');
> Step 5 : Execute the following query
> select * from ct1 c1, ct2 c2 where COALESCE(a1,b1)=a2;
> {code}
> ACTUAL RESULT:
> {code}
> The query returns nothing;
> {code}
> EXPECTED RESULT:
> {code}
> 1 NULL1
> {code}
> The issue seems to be because of the incorrect query plan. In the plan we can
> see:
> predicate:(a1 is not null and b1 is not null)
> which does not look correct. As a result, it is filtering out all the rows is
> any column mentioned in the COALESCE has null value.
> Please find the query plan below:
> {code}
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Map 1
> File Output Operator [FS_10]
> Map Join Operator [MAPJOIN_15] (rows=1 width=4)
>
> Conds:SEL_2.COALESCE(_col0,_col1)=RS_7._col0(Inner),HybridGraceHashJoin:true,Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE]
> BROADCAST [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_5] (rows=1 width=1)
> Output:["_col0"]
> Filter Operator [FIL_14] (rows=1 width=1)
> predicate:a2 is not null
> TableScan [TS_3] (rows=1 width=1)
> default@ct2,c2,Tbl:COMPLETE,Col:NONE,Output:["a2"]
> <-Select Operator [SEL_2] (rows=1 width=4)
> Output:["_col0","_col1"]
> Filter Operator [FIL_13] (rows=1 width=4)
> predicate:(a1 is not null and b1 is not null)
> TableScan [TS_0] (rows=1 width=4)
> default@ct1,c1,Tbl:COMPLETE,Col:NONE,Output:["a1","b1"]
> {code}
> This happens only if join is inner type, otherwise HiveJoinAddNotRule which
> creates this problem is skipped.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[
https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117883#comment-16117883
]
Hive QA commented on HIVE-17235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880774/HIVE-17235.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10995 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hadoop.hive.serde2.io.TestHiveDecimalWritableVersion.testVerifyHiveDecimalWritablePublicMethodsAndFieldsVersions
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6289/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6289/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6289/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880774 - PreCommit-HIVE-Build
> Add ORC Decimal64 Serialization/Deserialization (Part 1)
>
>
> Key: HIVE-17235
> URL: https://issues.apache.org/jira/browse/HIVE-17235
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch,
> HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch
>
>
> The storage-api changes for ORC-209.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17234) Remove HBase metastore from master
[ https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117879#comment-16117879 ] Lefty Leverenz commented on HIVE-17234: --- Doc note: This removes 15 hive.metastore.hbase.* configs from HiveConf.java. (Why wasn't *hive.metastore.hbase.file.metadata.threads* removed?) Most of them haven't been documented in the wiki yet. The section "Hive Metastore HBase" in Configuration Properties only has 3 configs: * [hive.metastore.hbase.cache.size | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.hbase.cache.size] (_still in the code_) * [hive.metastore.hbase.cache.ttl | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.hbase.cache.ttl] (_removed by this patch, so need to update wiki_) * [hive.metastore.hbase.cache.size | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.hbase.cache.size] (_removed by HIVE-9693 before the 2.0.0 release, so didn't belong in the wiki in the first place_) Here's the complete list, just for the record: * hive.metastore.hbase.catalog.cache.size * hive.metastore.hbase.aggregate.stats.cache.size * hive.metastore.hbase.aggregate.stats.max.partitions * hive.metastore.hbase.aggregate.stats.false.positive.probability * hive.metastore.hbase.aggregate.stats.max.variance * hive.metastore.hbase.cache.ttl (_documented in wiki_) * hive.metastore.hbase.cache.max.writer.wait * hive.metastore.hbase.cache.max.reader.wait * hive.metastore.hbase.cache.max.full * hive.metastore.hbase.cache.clean.until * hive.metastore.hbase.connection.class * hive.metastore.hbase.aggr.stats.cache.entries * hive.metastore.hbase.aggr.stats.memory.ttl * hive.metastore.hbase.aggr.stats.invalidator.frequency * hive.metastore.hbase.aggr.stats.hbase.ttl * hive.metastore.hbase.file.metadata.threads (_documented in wiki; not removed here_) > Remove HBase metastore from master > -- > > Key: HIVE-17234 > URL: https://issues.apache.org/jira/browse/HIVE-17234 > Project: Hive > Issue Type: Task > Components: HBase Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 3.0.0 > > Attachments: HIVE-17234.patch > > > No new development has been done on the HBase metastore in at least a year, > and to my knowledge no one is using it (nor is it even in a state to be fully > usable). Given the lack of interest in continuing to develop it, we should > remove it rather than leave dead code hanging around and extra tests taking > up time in test runs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117875#comment-16117875 ] Gopal V commented on HIVE-17235: [~owen.omalley]: the LongColumnVector clone needs re-implementation of all Decimal operators to be useful. Right now, that change adds an extra interface to test/debug with no actual operators to use for the end to end processing. I like the idea of a fast-path, but I'd rather not tackle the schema evolution problems right away (i.e the planner thinks it wants a Decimal(25,4), but the data is a Decimal(18,2) etc). We've never had that problem before where there are potentially two different vector column types for the same underlying column, which complicates things with no perceivable gains right now. I've moved the patch over to HIVE-17266 so that we can tackle it in a later iteration. > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17266) DecimalColumnVector64: Scaled fixed point column vector format
[
https://issues.apache.org/jira/browse/HIVE-17266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-17266:
---
Attachment: HIVE-17266.patch
> DecimalColumnVector64: Scaled fixed point column vector format
> --
>
> Key: HIVE-17266
> URL: https://issues.apache.org/jira/browse/HIVE-17266
> Project: Hive
> Issue Type: New Feature
> Components: storage-api
>Affects Versions: 3.0.0
>Reporter: Gopal V
> Attachments: HIVE-17266.patch
>
>
> I think we should make a new type that looks like:
> class Decimal64ColumnVector extends ColumnVector {
> long[] vector;
> int precision;
> int scale;
> }
> It will be extremely fast and provide a fast conduit to ORC.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16896) move replication load related work in semantic analysis phase to execution phase using a task
[
https://issues.apache.org/jira/browse/HIVE-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117837#comment-16117837
]
anishek commented on HIVE-16896:
Thanks [~sankarh] for the review. [~thejas]/[~daijy] please commit the patch.
> move replication load related work in semantic analysis phase to execution
> phase using a task
> -
>
> Key: HIVE-16896
> URL: https://issues.apache.org/jira/browse/HIVE-16896
> Project: Hive
> Issue Type: Sub-task
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16896.1.patch, HIVE-16896.2.patch,
> HIVE-16896.3.patch
>
>
> we want to not create too many tasks in memory in the analysis phase while
> loading data. Currently we load all the files in the bootstrap dump location
> as {{FileStatus[]}} and then iterate over it to load objects, we should
> rather move to
> {code}
> org.apache.hadoop.fs.RemoteIteratorlistFiles(Path
> f, boolean recursive)
> {code}
> which would internally batch and return values.
> additionally since we cant hand off partial tasks from analysis pahse =>
> execution phase, we are going to move the whole repl load functionality to
> execution phase so we can better control creation/execution of tasks (not
> related to hive {{Task}}, we may get rid of ReplCopyTask)
> Additional consideration to take into account at the end of this jira is to
> see if we want to specifically do a multi threaded load of bootstrap dump.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15705) Event replication for constraints
[
https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117834#comment-16117834
]
Hive QA commented on HIVE-15705:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880735/HIVE-15705.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth
(batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
(batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=241)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6288/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6288/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880735 - PreCommit-HIVE-Build
> Event replication for constraints
> -
>
> Key: HIVE-15705
> URL: https://issues.apache.org/jira/browse/HIVE-15705
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15705.1.patch, HIVE-15705.2.patch,
> HIVE-15705.3.patch, HIVE-15705.4.patch, HIVE-15705.5.patch, HIVE-15705.6.patch
>
>
> Make event replication for primary key and foreign key work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17247) HoS DPP: UDFs on the partition column side does not evaluate correctly
[
https://issues.apache.org/jira/browse/HIVE-17247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17247:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the review [~lirui]. Merged to master.
> HoS DPP: UDFs on the partition column side does not evaluate correctly
> --
>
> Key: HIVE-17247
> URL: https://issues.apache.org/jira/browse/HIVE-17247
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-17247.1.patch
>
>
> Same problem as HIVE-12473 and HIVE-12667.
> The query below (uses tables from {{spark_dynamic_partition_pruning.q}})
> returns incorrect results:
> {code}
> select count(*) from srcpart join srcpart_date on (day(srcpart.ds) =
> day(srcpart_date.ds)) where srcpart_date.`date` = '2008-04-08';
> {code}
> It returns a value of 0 when DPP is on, when it is disabled it returns 1000
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117792#comment-16117792 ] Lefty Leverenz commented on HIVE-8472: -- The documentation belongs here: * [LanguageManual -- DDL -- Alter Database | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterDatabase] > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0, 3.0.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15705) Event replication for constraints
[
https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117787#comment-16117787
]
Hive QA commented on HIVE-15705:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880735/HIVE-15705.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6287/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6287/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880735 - PreCommit-HIVE-Build
> Event replication for constraints
> -
>
> Key: HIVE-15705
> URL: https://issues.apache.org/jira/browse/HIVE-15705
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15705.1.patch, HIVE-15705.2.patch,
> HIVE-15705.3.patch, HIVE-15705.4.patch, HIVE-15705.5.patch, HIVE-15705.6.patch
>
>
> Make event replication for primary key and foreign key work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Attachment: HIVE-17235.06.patch > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Attachment: (was: HIVE-17235.06.patch) > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[
https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117744#comment-16117744
]
Hive QA commented on HIVE-8472:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880715/HIVE-8472.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10998 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6286/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6286/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6286/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880715 - PreCommit-HIVE-Build
> Add ALTER DATABASE SET LOCATION
> ---
>
> Key: HIVE-8472
> URL: https://issues.apache.org/jira/browse/HIVE-8472
> Project: Hive
> Issue Type: Improvement
> Components: Database/Schema
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Jeremy Beard
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch
>
>
> Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there
> was an equivalent for databases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17265: --- Attachment: (was: HIVE-17265.patch) > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-17265.patch > > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17265: --- Attachment: HIVE-17265.patch > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-17265.patch > > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17265: --- Attachment: HIVE-17265.patch > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-17265.patch > > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17265: --- Status: Patch Available (was: In Progress) > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17265 started by Jesus Camacho Rodriguez. -- > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17265) Cache merged column stats from retrieved partitions
[ https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-17265: -- > Cache merged column stats from retrieved partitions > --- > > Key: HIVE-17265 > URL: https://issues.apache.org/jira/browse/HIVE-17265 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Currently when we retrieve stats from the metastore for a column in a > partitioned table, we will execute the logic to merge the column stats coming > from each partition multiple times. > Even though we avoid multiple calls to metastore if the cache for the stats > in enabled, merging the stats for a given column can take a large amount of > time if there is a large number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17203) Add InterfaceAudience and InterfaceStability annotations for HCat APIs
[
https://issues.apache.org/jira/browse/HIVE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117692#comment-16117692
]
Hive QA commented on HIVE-17203:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880708/HIVE-17203.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6285/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6285/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880708 - PreCommit-HIVE-Build
> Add InterfaceAudience and InterfaceStability annotations for HCat APIs
> --
>
> Key: HIVE-17203
> URL: https://issues.apache.org/jira/browse/HIVE-17203
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17203.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17111) TestSparkCliDriver does not use LocalHiveSparkClient
[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117628#comment-16117628
]
Xuefu Zhang commented on HIVE-17111:
My take is that local is for dev, local-cluster for precommit test, and
miniYarnCluster is for mimicking production. I'm not sure how much it saves by
switching to LocalHiveSparkClient. If that's a problem, we can consider running
multiple tests with the same local-cluster.
> TestSparkCliDriver does not use LocalHiveSparkClient
>
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16811) Estimate statistics in absence of stats
[
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16811:
---
Attachment: HIVE-16811.6.patch
> Estimate statistics in absence of stats
> ---
>
> Key: HIVE-16811
> URL: https://issues.apache.org/jira/browse/HIVE-16811
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16811.1.patch, HIVE-16811.2.patch,
> HIVE-16811.3.patch, HIVE-16811.4.patch, HIVE-16811.5.patch, HIVE-16811.6.patch
>
>
> Currently Join ordering completely bails out in absence of statistics and
> this could lead to bad joins such as cross joins.
> e.g. following select query will produce cross join.
> {code:sql}
> create table supplier (S_SUPPKEY INT, S_NAME STRING, S_ADDRESS STRING,
> S_NATIONKEY INT,
> S_PHONE STRING, S_ACCTBAL DOUBLE, S_COMMENT STRING)
> CREATE TABLE lineitem (L_ORDERKEY INT,
> L_PARTKEY INT,
> L_SUPPKEY INT,
> L_LINENUMBERINT,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAX DOUBLE,
> L_RETURNFLAGSTRING,
> L_LINESTATUSSTRING,
> l_shipdate STRING,
> L_COMMITDATESTRING,
> L_RECEIPTDATE STRING,
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENT STRING) partitioned by (dl
> int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|';
> CREATE TABLE part(
> p_partkey INT,
> p_name STRING,
> p_mfgr STRING,
> p_brand STRING,
> p_type STRING,
> p_size INT,
> p_container STRING,
> p_retailprice DOUBLE,
> p_comment STRING
> );
> explain select count(1) from part,supplier,lineitem where p_partkey =
> l_partkey and s_suppkey = l_suppkey;
> {code}
> Estimating stats will prevent join ordering algorithm to bail out and come up
> with join at least better than cross join
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16811) Estimate statistics in absence of stats
[
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16811:
---
Status: Patch Available (was: Open)
> Estimate statistics in absence of stats
> ---
>
> Key: HIVE-16811
> URL: https://issues.apache.org/jira/browse/HIVE-16811
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16811.1.patch, HIVE-16811.2.patch,
> HIVE-16811.3.patch, HIVE-16811.4.patch, HIVE-16811.5.patch, HIVE-16811.6.patch
>
>
> Currently Join ordering completely bails out in absence of statistics and
> this could lead to bad joins such as cross joins.
> e.g. following select query will produce cross join.
> {code:sql}
> create table supplier (S_SUPPKEY INT, S_NAME STRING, S_ADDRESS STRING,
> S_NATIONKEY INT,
> S_PHONE STRING, S_ACCTBAL DOUBLE, S_COMMENT STRING)
> CREATE TABLE lineitem (L_ORDERKEY INT,
> L_PARTKEY INT,
> L_SUPPKEY INT,
> L_LINENUMBERINT,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAX DOUBLE,
> L_RETURNFLAGSTRING,
> L_LINESTATUSSTRING,
> l_shipdate STRING,
> L_COMMITDATESTRING,
> L_RECEIPTDATE STRING,
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENT STRING) partitioned by (dl
> int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|';
> CREATE TABLE part(
> p_partkey INT,
> p_name STRING,
> p_mfgr STRING,
> p_brand STRING,
> p_type STRING,
> p_size INT,
> p_container STRING,
> p_retailprice DOUBLE,
> p_comment STRING
> );
> explain select count(1) from part,supplier,lineitem where p_partkey =
> l_partkey and s_suppkey = l_suppkey;
> {code}
> Estimating stats will prevent join ordering algorithm to bail out and come up
> with join at least better than cross join
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16811) Estimate statistics in absence of stats
[
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16811:
---
Status: Open (was: Patch Available)
> Estimate statistics in absence of stats
> ---
>
> Key: HIVE-16811
> URL: https://issues.apache.org/jira/browse/HIVE-16811
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16811.1.patch, HIVE-16811.2.patch,
> HIVE-16811.3.patch, HIVE-16811.4.patch, HIVE-16811.5.patch
>
>
> Currently Join ordering completely bails out in absence of statistics and
> this could lead to bad joins such as cross joins.
> e.g. following select query will produce cross join.
> {code:sql}
> create table supplier (S_SUPPKEY INT, S_NAME STRING, S_ADDRESS STRING,
> S_NATIONKEY INT,
> S_PHONE STRING, S_ACCTBAL DOUBLE, S_COMMENT STRING)
> CREATE TABLE lineitem (L_ORDERKEY INT,
> L_PARTKEY INT,
> L_SUPPKEY INT,
> L_LINENUMBERINT,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAX DOUBLE,
> L_RETURNFLAGSTRING,
> L_LINESTATUSSTRING,
> l_shipdate STRING,
> L_COMMITDATESTRING,
> L_RECEIPTDATE STRING,
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENT STRING) partitioned by (dl
> int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|';
> CREATE TABLE part(
> p_partkey INT,
> p_name STRING,
> p_mfgr STRING,
> p_brand STRING,
> p_type STRING,
> p_size INT,
> p_container STRING,
> p_retailprice DOUBLE,
> p_comment STRING
> );
> explain select count(1) from part,supplier,lineitem where p_partkey =
> l_partkey and s_suppkey = l_suppkey;
> {code}
> Estimating stats will prevent join ordering algorithm to bail out and come up
> with join at least better than cross join
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Attachment: HIVE-17235.06.patch > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Attachment: (was: HIVE-17235.06.patch) > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15767) Hive On Spark is not working on secure clusters from Oozie
[
https://issues.apache.org/jira/browse/HIVE-15767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117614#comment-16117614
]
Xuefu Zhang commented on HIVE-15767:
>From what I see, the patch seems logical, harmless at least. What I don't
>understand is that why Spark would attempt reading this file. As a side note,
>I didn't find the source in Spark code base that does this.
> Hive On Spark is not working on secure clusters from Oozie
> --
>
> Key: HIVE-15767
> URL: https://issues.apache.org/jira/browse/HIVE-15767
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.1.1
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: HIVE-15767-001.patch, HIVE-15767-002.patch,
> HIVE-15767.1.patch
>
>
> When a HiveAction is launched form Oozie with Hive On Spark enabled, we're
> getting errors:
> {noformat}
> Caused by: java.io.IOException: Exception reading
> file:/yarn/nm/usercache/yshi/appcache/application_1485271416004_0022/container_1485271416004_0022_01_02/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:188)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:155)
> {noformat}
> This is caused by passing the {{mapreduce.job.credentials.binary}} property
> to the Spark configuration in RemoteHiveSparkClient.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[
https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117611#comment-16117611
]
Hive QA commented on HIVE-8472:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880715/HIVE-8472.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10998 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6284/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6284/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880715 - PreCommit-HIVE-Build
> Add ALTER DATABASE SET LOCATION
> ---
>
> Key: HIVE-8472
> URL: https://issues.apache.org/jira/browse/HIVE-8472
> Project: Hive
> Issue Type: Improvement
> Components: Database/Schema
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Jeremy Beard
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch
>
>
> Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there
> was an equivalent for databases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Status: Patch Available (was: Open) > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117610#comment-16117610 ] Matt McCline commented on HIVE-17235: - Added "Part 1" to the title to keep this JIRA focused. DecimalColumnVector64 can be added in a subsequent change. > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Summary: Add ORC Decimal64 Serialization/Deserialization (Part 1) (was: Add ORC Decimal64 Serialization/Deserialization) > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Attachment: HIVE-17235.06.patch > Add ORC Decimal64 Serialization/Deserialization > --- > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions
[ https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16895: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Patch pushed to master. > Multi-threaded execution of bootstrap dump of partitions > - > > Key: HIVE-16895 > URL: https://issues.apache.org/jira/browse/HIVE-16895 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16895.1.patch, HIVE-16895.2.patch > > > to allow faster execution of bootstrap dump phase we dump multiple partitions > from same table simultaneously. > even though dumping functions is not going to be a blocker, moving to > similar execution modes for all metastore objects will make code more > coherent. > Bootstrap dump at db level does : > * boostrap of all tables > ** boostrap of all partitions in a table. (scope of current jira) > * boostrap of all functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16758) Better Select Number of Replications
[
https://issues.apache.org/jira/browse/HIVE-16758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117568#comment-16117568
]
Chao Sun commented on HIVE-16758:
-
+1 on patch v3.
> Better Select Number of Replications
>
>
> Key: HIVE-16758
> URL: https://issues.apache.org/jira/browse/HIVE-16758
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-16758.1.patch, HIVE-16758.2.patch,
> HIVE-16758.3.patch
>
>
> {{org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.java}}
> We should be smarter about how we pick a replication number. We should add a
> new configuration equivalent to {{mapreduce.client.submit.file.replication}}.
> This value should be around the square root of the number of nodes and not
> hard-coded in the code.
> {code}
> public static final String DFS_REPLICATION_MAX = "dfs.replication.max";
> private int minReplication = 10;
> @Override
> protected void initializeOp(Configuration hconf) throws HiveException {
> ...
> int dfsMaxReplication = hconf.getInt(DFS_REPLICATION_MAX, minReplication);
> // minReplication value should not cross the value of dfs.replication.max
> minReplication = Math.min(minReplication, dfsMaxReplication);
> }
> {code}
> https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117565#comment-16117565
]
Chao Sun commented on HIVE-17257:
-
[~kellyzly]: the empty files maybe generated if the result set is empty and if
you have multiple mapper/reducers with file sink. Example:
{code}
set hive.execution.engine=spark;
set hive.auto.convert.join=false;
set mapreduce.job.reduces=1000;
create table dummy (a string);
insert overwrite directory '/tmp/test' select src.key from src join dummy on
src.key = dummy.a;
{code}
The above will generate 1000 empty files in /tmp/test.
[~xuefuz]: I need to revise the patch. There's an issue where HoS won't launch
task for the final merge job since the input data is empty.
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15705) Event replication for constraints
[ https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15705: -- Attachment: HIVE-15705.6.patch > Event replication for constraints > - > > Key: HIVE-15705 > URL: https://issues.apache.org/jira/browse/HIVE-15705 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15705.1.patch, HIVE-15705.2.patch, > HIVE-15705.3.patch, HIVE-15705.4.patch, HIVE-15705.5.patch, HIVE-15705.6.patch > > > Make event replication for primary key and foreign key work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15705) Event replication for constraints
[ https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15705: -- Attachment: (was: HIVE-15705.6.patch) > Event replication for constraints > - > > Key: HIVE-15705 > URL: https://issues.apache.org/jira/browse/HIVE-15705 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15705.1.patch, HIVE-15705.2.patch, > HIVE-15705.3.patch, HIVE-15705.4.patch, HIVE-15705.5.patch > > > Make event replication for primary key and foreign key work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15705) Event replication for constraints
[ https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117559#comment-16117559 ] Daniel Dai commented on HIVE-15705: --- Resync with master. Also skip review#9 from the patch. constraintName cannot be null. It originally works assuming constraintName is unchanged if seq>1. > Event replication for constraints > - > > Key: HIVE-15705 > URL: https://issues.apache.org/jira/browse/HIVE-15705 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15705.1.patch, HIVE-15705.2.patch, > HIVE-15705.3.patch, HIVE-15705.4.patch, HIVE-15705.5.patch, HIVE-15705.6.patch > > > Make event replication for primary key and foreign key work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15705) Event replication for constraints
[ https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15705: -- Attachment: HIVE-15705.6.patch > Event replication for constraints > - > > Key: HIVE-15705 > URL: https://issues.apache.org/jira/browse/HIVE-15705 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15705.1.patch, HIVE-15705.2.patch, > HIVE-15705.3.patch, HIVE-15705.4.patch, HIVE-15705.5.patch, HIVE-15705.6.patch > > > Make event replication for primary key and foreign key work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17008) DbNotificationListener should skip failed events
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Burkert reassigned HIVE-17008:
--
Assignee: (was: Dan Burkert)
> DbNotificationListener should skip failed events
>
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) DbNotificationListener should skip failed events
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117546#comment-16117546
]
Dan Burkert commented on HIVE-17008:
Hi all, I'd like to pass this off to someone with more experience getting Hive
patches merged.
> DbNotificationListener should skip failed events
>
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16925) isSlowStart lost during refactoring
[
https://issues.apache.org/jira/browse/HIVE-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117539#comment-16117539
]
Hive QA commented on HIVE-16925:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880703/HIVE-16925.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6283/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6283/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6283/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880703 - PreCommit-HIVE-Build
> isSlowStart lost during refactoring
> ---
>
> Key: HIVE-16925
> URL: https://issues.apache.org/jira/browse/HIVE-16925
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-16925.patch
>
>
> TezEdgeProperty.setAutoReduce() should have isSlowStart as parameter
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17172) add ordering checks to DiskRangeList
[ https://issues.apache.org/jira/browse/HIVE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117503#comment-16117503 ] Prasanth Jayachandran commented on HIVE-17172: -- +1 > add ordering checks to DiskRangeList > > > Key: HIVE-17172 > URL: https://issues.apache.org/jira/browse/HIVE-17172 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17172.01.patch, HIVE-17172.02.patch, > HIVE-17172.ADDENDUM.patch, HIVE-17172.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17172) add ordering checks to DiskRangeList
[ https://issues.apache.org/jira/browse/HIVE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117442#comment-16117442 ] Sergey Shelukhin commented on HIVE-17172: - [~q79969786] thanks for catching this! > add ordering checks to DiskRangeList > > > Key: HIVE-17172 > URL: https://issues.apache.org/jira/browse/HIVE-17172 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17172.01.patch, HIVE-17172.02.patch, > HIVE-17172.ADDENDUM.patch, HIVE-17172.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17172) add ordering checks to DiskRangeList
[ https://issues.apache.org/jira/browse/HIVE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17172: Attachment: HIVE-17172.ADDENDUM.patch [~prasanth_j] can you please take a look? a tiny addendum patch > add ordering checks to DiskRangeList > > > Key: HIVE-17172 > URL: https://issues.apache.org/jira/browse/HIVE-17172 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17172.01.patch, HIVE-17172.02.patch, > HIVE-17172.ADDENDUM.patch, HIVE-17172.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17258) Incorrect log messages in the Hive.java
[
https://issues.apache.org/jira/browse/HIVE-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117415#comment-16117415
]
Hive QA commented on HIVE-17258:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880701/HIVE-17258.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.beeline.hs2connection.TestBeelineConnectionUsingHiveSite.testBeelineConnectionHttp
(batchId=222)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentStatements (batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6282/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6282/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880701 - PreCommit-HIVE-Build
> Incorrect log messages in the Hive.java
> ---
>
> Key: HIVE-17258
> URL: https://issues.apache.org/jira/browse/HIVE-17258
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-17258.patch
>
>
> There are couple of typos when using LOG methods in the
> org.apache.hadoop.hive.ql.metadata.Hive class.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17264) BaseSemanticAnalyzer.escapeSQLString escapes \0 unsafely
[ https://issues.apache.org/jira/browse/HIVE-17264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Santangelo updated HIVE-17264: Affects Version/s: 2.3.0 > BaseSemanticAnalyzer.escapeSQLString escapes \0 unsafely > > > Key: HIVE-17264 > URL: https://issues.apache.org/jira/browse/HIVE-17264 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 2.3.0 >Reporter: Jesse Santangelo >Priority: Minor > > Currently escapeSQLString() converts the null character to \0. If there are > numerical values after the null character, calling unescapeSQLString() can > treat the resulting value as an octal literal. > Example: > "FOOBAR_EVENT\u2017" > unescapeSQLString converts \u to the null character. Calling > escapeSQLString() on that resulting string currently gives > "FOOBAR_EVENT\02017" > A third call of unescapeSQLString() will then see \020 and convert it to 0x10. > This is an issue for the Hbase connector where key values are processed twice > (I'm unfamiliar with the hive codebase so I don't know why) and so the > connector gets the wrong key value if it's formatted like the example. > In my version I've changed escapeSQLString to replace the null character with > '\u' instead of '\0' and that seems to have fixed the issue, but I don't > know if that might break anything else. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8472: --- Affects Version/s: 3.0.0 > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0, 3.0.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8472: --- Status: Patch Available (was: Open) > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8472: --- Attachment: HIVE-8472.3.patch > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8472: --- Attachment: (was: HIVE-8472.3.patch) > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8472: --- Status: Open (was: Patch Available) > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17203) Add InterfaceAudience and InterfaceStability annotations for HCat APIs
[ https://issues.apache.org/jira/browse/HIVE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117315#comment-16117315 ] Sahil Takiar commented on HIVE-17203: - My best attempt at updating the tags of the HCat APIs. I intentionally avoided features that look like they aren't used. > Add InterfaceAudience and InterfaceStability annotations for HCat APIs > -- > > Key: HIVE-17203 > URL: https://issues.apache.org/jira/browse/HIVE-17203 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 3.0.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17203.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17203) Add InterfaceAudience and InterfaceStability annotations for HCat APIs
[ https://issues.apache.org/jira/browse/HIVE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17203: Status: Patch Available (was: Open) > Add InterfaceAudience and InterfaceStability annotations for HCat APIs > -- > > Key: HIVE-17203 > URL: https://issues.apache.org/jira/browse/HIVE-17203 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 3.0.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17203.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17203) Add InterfaceAudience and InterfaceStability annotations for HCat APIs
[ https://issues.apache.org/jira/browse/HIVE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17203: Attachment: HIVE-17203.1.patch > Add InterfaceAudience and InterfaceStability annotations for HCat APIs > -- > > Key: HIVE-17203 > URL: https://issues.apache.org/jira/browse/HIVE-17203 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 3.0.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17203.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8472) Add ALTER DATABASE SET LOCATION
[ https://issues.apache.org/jira/browse/HIVE-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-8472: --- Attachment: HIVE-8472.3.patch Thank you, [~alangates]. I've made the changes you suggested. [~leftylev], where might one make the documentation changes that [~alangates] mentions? > Add ALTER DATABASE SET LOCATION > --- > > Key: HIVE-8472 > URL: https://issues.apache.org/jira/browse/HIVE-8472 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0 >Reporter: Jeremy Beard >Assignee: Mithun Radhakrishnan > Attachments: HIVE-8472.1.patch, HIVE-8472.3.patch > > > Similarly to ALTER TABLE tablename SET LOCATION, it would be helpful if there > was an equivalent for databases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16925) isSlowStart lost during refactoring
[ https://issues.apache.org/jira/browse/HIVE-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Danilov updated HIVE-16925: Status: Patch Available (was: Open) > isSlowStart lost during refactoring > --- > > Key: HIVE-16925 > URL: https://issues.apache.org/jira/browse/HIVE-16925 > Project: Hive > Issue Type: Bug >Reporter: Oleg Danilov >Priority: Minor > Attachments: HIVE-16925.patch > > > TezEdgeProperty.setAutoReduce() should have isSlowStart as parameter -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16925) isSlowStart lost during refactoring
[ https://issues.apache.org/jira/browse/HIVE-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Danilov updated HIVE-16925: Attachment: HIVE-16925.patch > isSlowStart lost during refactoring > --- > > Key: HIVE-16925 > URL: https://issues.apache.org/jira/browse/HIVE-16925 > Project: Hive > Issue Type: Bug >Reporter: Oleg Danilov >Priority: Minor > Attachments: HIVE-16925.patch > > > TezEdgeProperty.setAutoReduce() should have isSlowStart as parameter -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17258) Incorrect log messages in the Hive.java
[ https://issues.apache.org/jira/browse/HIVE-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Danilov updated HIVE-17258: Status: Patch Available (was: Open) > Incorrect log messages in the Hive.java > --- > > Key: HIVE-17258 > URL: https://issues.apache.org/jira/browse/HIVE-17258 > Project: Hive > Issue Type: Bug >Reporter: Oleg Danilov >Priority: Trivial > Attachments: HIVE-17258.patch > > > There are couple of typos when using LOG methods in the > org.apache.hadoop.hive.ql.metadata.Hive class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17258) Incorrect log messages in the Hive.java
[ https://issues.apache.org/jira/browse/HIVE-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Danilov updated HIVE-17258: Attachment: HIVE-17258.patch Fixed typos > Incorrect log messages in the Hive.java > --- > > Key: HIVE-17258 > URL: https://issues.apache.org/jira/browse/HIVE-17258 > Project: Hive > Issue Type: Bug >Reporter: Oleg Danilov >Priority: Trivial > Attachments: HIVE-17258.patch > > > There are couple of typos when using LOG methods in the > org.apache.hadoop.hive.ql.metadata.Hive class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17172) add ordering checks to DiskRangeList
[ https://issues.apache.org/jira/browse/HIVE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17172: Priority: Blocker (was: Major) > add ordering checks to DiskRangeList > > > Key: HIVE-17172 > URL: https://issues.apache.org/jira/browse/HIVE-17172 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17172.01.patch, HIVE-17172.02.patch, > HIVE-17172.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17172) add ordering checks to DiskRangeList
[ https://issues.apache.org/jira/browse/HIVE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117226#comment-16117226 ] Sergey Shelukhin commented on HIVE-17172: - Hmm, I forgot the special case of the mutation helper. Would need to fix before release > add ordering checks to DiskRangeList > > > Key: HIVE-17172 > URL: https://issues.apache.org/jira/browse/HIVE-17172 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17172.01.patch, HIVE-17172.02.patch, > HIVE-17172.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (HIVE-17172) add ordering checks to DiskRangeList
[ https://issues.apache.org/jira/browse/HIVE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HIVE-17172: - > add ordering checks to DiskRangeList > > > Key: HIVE-17172 > URL: https://issues.apache.org/jira/browse/HIVE-17172 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17172.01.patch, HIVE-17172.02.patch, > HIVE-17172.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[
https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117193#comment-16117193
]
Sergey Shelukhin commented on HIVE-17126:
-
Actually I guess I see the query from direct SQL and the issue is not related
to the text of the query.
"Unknown system variable 'OPTION'"
Note that direct SQL calls {noformat}SET
@@session.sql_mode=ANSI_QUOTES{noformat} on mysql to use double quotes
correctly on MySQL and avoid the weird mysql qoutes.
You can disable direct SQL and see if it fixes the issue since DN itself uses
the backticks afaik.
However, from the error message it doesn't look related to quotes.
> Hive Metastore is incompatible with MariaDB 10.x
>
>
> Key: HIVE-17126
> URL: https://issues.apache.org/jira/browse/HIVE-17126
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0, 2.0.0
>Reporter: Eric Yang
>
> MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage
> of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as
> highly available metastore. Datanucleus generate SQL statements that are not
> parsable by MariaDB 10.x when dropping Hive table or database schema.
> Without MariaDB HA setup, the SQL statement problem also exists for metastore
> interaction with MariaDB 10.x.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[
https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117158#comment-16117158
]
Todd Richmond commented on HIVE-16529:
--
sorry - libpam4j in public repos - not JPAM (which does not have a corruption
patch)
> Replace JPAM with libpam4j for PAM authentication
> -
>
> Key: HIVE-16529
> URL: https://issues.apache.org/jira/browse/HIVE-16529
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 1.2.0
>Reporter: Richard Ding
>Assignee: Sailaja Navvluru
>
> PAM authentication is an important feature available since Hive 0.13. But
> Hive blog gives the following warnings:
> {quote}
> JPAM library that is used to provide the PAM authentication mode can cause
> HiveServer2 to go down if a user's password has expired. This happens because
> of segfault/core dumps from native code invoked by JPAM. Some users have also
> reported crashes during logins in other cases as well. Use of LDAP or
> KERBEROS is recommended.
> {quote}
> JPAM also requires user to install a native library. Furthermore, JPAM
> library seems not to have been updated since 2007.
> Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j
> which doesn't require installation of native library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[
https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117120#comment-16117120
]
Hive QA commented on HIVE-17181:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880670/HIVE-17181.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6281/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6281/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6281/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880670 - PreCommit-HIVE-Build
> HCatOutputFormat should expose complete output-schema (including
> partition-keys) for dynamic-partitioning MR jobs
> -
>
> Key: HIVE-17181
> URL: https://issues.apache.org/jira/browse/HIVE-17181
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch,
> HIVE-17181.3.patch, HIVE-17181.branch-2.patch
>
>
> Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic
> partitioning are expected to call the following API methods:
> # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to
> write to. This call populates the {{OutputJobInfo}} with details fetched from
> the Metastore.
> # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data
> being written.
> It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows:
> {code:java}
> HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf));
> {code}
> Unfortunately, {{getTableSchema()}} returns only the record-schema, not the
> entire table's schema. We'll need a better API for use in M/R jobs to get the
> complete table-schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17027) HPL/SQL requires single quotes for string literals, resulting in surprising behavior
[
https://issues.apache.org/jira/browse/HIVE-17027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117115#comment-16117115
]
Shawn Weeks commented on HIVE-17027:
>From an Oracle perspective double and single quotes mean very different things
>and treating them the same would lead to unpredictable behavior. Not sure how
>MySQL and Postgres handle this but in Oracle double quotes are used to quote
>object names with special characters while Hive uses backticks to do the same
>thing.
> HPL/SQL requires single quotes for string literals, resulting in surprising
> behavior
>
>
> Key: HIVE-17027
> URL: https://issues.apache.org/jira/browse/HIVE-17027
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: Carter Shanklin
>Priority: Critical
>
> This bug is part of a series of issues and surprising behavior I encountered
> writing a reporting script that would aggregate values and give rows
> different classifications based on an the aggregate. Addressing some or all
> of these issues would make HPL/SQL more accessible to newcomers.
> Consider this script:
> {code}
> CREATE FUNCTION test1()
> RETURNS STRING
> DECLARE
> VAR ret string;
> BEGIN
> ret := 'VALUE IS SET';
> print(ret);
> END;
> CREATE FUNCTION test2()
> RETURNS STRING
> DECLARE
> VAR ret string;
> BEGIN
> ret := "VALUE IS SET";
> print(ret);
> END;
> test1();
> test2();
> {code}
> The output of this script is:
> VALUE IS SET
> ret
> Hive accepts both quoting styles. It would be better if HPL/SQL did as well,
> or threw an error for th
> e unsupported style.
> Version = 3.0.0-SNAPSHOT r71f52d8ad512904b3f2c4f04fe39a33f2834f1f2
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[
https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117096#comment-16117096
]
Thejas M Nair edited comment on HIVE-16529 at 8/7/17 7:21 PM:
--
Thanks for the heads up [~trichmond]!
bq. However, the version of JPAM on public repos will also cause corruption
unless a single patch has been applied.
Did you mean to say "libpam4j" not JPAM in the above sentence ?
was (Author: thejas):
bq. However, the version of JPAM on public repos will also cause corruption
unless a single patch has been applied.
Did you mean to say "libpam4j" not JPAM in the above sentence ?
> Replace JPAM with libpam4j for PAM authentication
> -
>
> Key: HIVE-16529
> URL: https://issues.apache.org/jira/browse/HIVE-16529
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 1.2.0
>Reporter: Richard Ding
>Assignee: Sailaja Navvluru
>
> PAM authentication is an important feature available since Hive 0.13. But
> Hive blog gives the following warnings:
> {quote}
> JPAM library that is used to provide the PAM authentication mode can cause
> HiveServer2 to go down if a user's password has expired. This happens because
> of segfault/core dumps from native code invoked by JPAM. Some users have also
> reported crashes during logins in other cases as well. Use of LDAP or
> KERBEROS is recommended.
> {quote}
> JPAM also requires user to install a native library. Furthermore, JPAM
> library seems not to have been updated since 2007.
> Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j
> which doesn't require installation of native library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[
https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117096#comment-16117096
]
Thejas M Nair edited comment on HIVE-16529 at 8/7/17 7:21 PM:
--
Thanks for the heads up [~todd_richmond]!
bq. However, the version of JPAM on public repos will also cause corruption
unless a single patch has been applied.
Did you mean to say "libpam4j" not JPAM in the above sentence ?
was (Author: thejas):
Thanks for the heads up [~trichmond]!
bq. However, the version of JPAM on public repos will also cause corruption
unless a single patch has been applied.
Did you mean to say "libpam4j" not JPAM in the above sentence ?
> Replace JPAM with libpam4j for PAM authentication
> -
>
> Key: HIVE-16529
> URL: https://issues.apache.org/jira/browse/HIVE-16529
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 1.2.0
>Reporter: Richard Ding
>Assignee: Sailaja Navvluru
>
> PAM authentication is an important feature available since Hive 0.13. But
> Hive blog gives the following warnings:
> {quote}
> JPAM library that is used to provide the PAM authentication mode can cause
> HiveServer2 to go down if a user's password has expired. This happens because
> of segfault/core dumps from native code invoked by JPAM. Some users have also
> reported crashes during logins in other cases as well. Use of LDAP or
> KERBEROS is recommended.
> {quote}
> JPAM also requires user to install a native library. Furthermore, JPAM
> library seems not to have been updated since 2007.
> Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j
> which doesn't require installation of native library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[
https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117096#comment-16117096
]
Thejas M Nair commented on HIVE-16529:
--
bq. However, the version of JPAM on public repos will also cause corruption
unless a single patch has been applied.
Did you mean to say "libpam4j" not JPAM in the above sentence ?
> Replace JPAM with libpam4j for PAM authentication
> -
>
> Key: HIVE-16529
> URL: https://issues.apache.org/jira/browse/HIVE-16529
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 1.2.0
>Reporter: Richard Ding
>Assignee: Sailaja Navvluru
>
> PAM authentication is an important feature available since Hive 0.13. But
> Hive blog gives the following warnings:
> {quote}
> JPAM library that is used to provide the PAM authentication mode can cause
> HiveServer2 to go down if a user's password has expired. This happens because
> of segfault/core dumps from native code invoked by JPAM. Some users have also
> reported crashes during logins in other cases as well. Use of LDAP or
> KERBEROS is recommended.
> {quote}
> JPAM also requires user to install a native library. Furthermore, JPAM
> library seems not to have been updated since 2007.
> Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j
> which doesn't require installation of native library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters
[
https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117076#comment-16117076
]
Sahil Takiar commented on HIVE-17237:
-
+1
> HMS wastes 26.4% of memory due to dup strings in
> metastore.api.Partition.parameters
> ---
>
> Key: HIVE-17237
> URL: https://issues.apache.org/jira/browse/HIVE-17237
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HIVE-17237.01.patch
>
>
> I've analyzed a heap dump from a production Hive installation using jxray
> (www.jxray.com) It turns out that there are a lot of duplicate strings in
> memory, that waste 26.4% of the heap. Most of them come from HashMaps
> referenced by org.apache.hadoop.hive.metastore.api.Partition.parameters.
> Below is the relevant section of the jxray report.
> Looking at Partition.java, I see that in the past somebody has already added
> code to intern keys and values in the parameters table when it's first set
> up. However, when more key-value pairs are added, they are not interned, and
> that probably explains the reason for all these duplicate strings. Also when
> a Partition instance is deserialized, no interning of parameters is currently
> done.
> {code}
> 6. DUPLICATE STRINGS
> Total strings: 3,273,557 Unique strings: 460,390 Duplicate values: 110,232
> Overhead: 3,220,458K (26.4%)
>
> ===
> 7. REFERENCE CHAINS FOR DUPLICATE STRINGS
> 2,326,150K (19.1%), 597058 dup strings (36386 unique), 597058 dup backing
> arrays:
> 39949 of "-1", 39088 of "true", 28959 of "8", 20987 of "1", 18437 of "10",
> 9583 of "9", 5908 of "269664", 5691 of "174528", 4598 of "133980", 4598 of
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length
> 3560]"
> ... and 419200 more strings, of which 36376 are unique
> Also contains one-char strings: 217 of "6", 147 of "7", 91 of "4", 28 of "5",
> 28 of "2", 21 of "0"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <--
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
> <-- Java Local
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
> [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
> 463,060K (3.8%), 119644 dup strings (34075 unique), 119644 dup backing
> arrays:
> 7914 of "true", 7912 of "-1", 6578 of "8", 5606 of "1", 2302 of "10", 1626 of
> "174528", 1223 of "9", 970 of "171680", 837 of "269664", 657 of "133980"
> ... and 84009 more strings, of which 34065 are unique
> Also contains one-char strings: 42 of "7", 31 of "6", 20 of "4", 8 of "5", 5
> of "2", 3 of "0"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.TreeMap}.values <-- Java Local (j.u.TreeMap) [@6f084afa0,@73aac9e68]
> 233,384K (1.9%), 64601 dup strings (27295 unique), 64601 dup backing arrays:
> 4472 of "true", 4173 of "-1", 3798 of "1", 3591 of "8", 813 of "174528", 684
> of "10" ... and 44568 more strings, of which 27285 are unique
> Also contains one-char strings: 305 of "7", 301 of "0", 277 of "4", 146 of
> "6", 29 of "2", 23 of "5", 19 of "9", 2 of "3"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <-- Java Local (j.u.ArrayList)
> [@4f4cfbd10,@536122408,@726616778]
> ...
> 52,916K (0.4%), 597058 dup strings (16 unique), 597058 dup backing arrays:
> <-- {j.u.HashMap}.keys <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <--
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
> <-- Java Local
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
> [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16529) Replace JPAM with libpam4j for PAM authentication
[
https://issues.apache.org/jira/browse/HIVE-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116997#comment-16116997
]
Todd Richmond commented on HIVE-16529:
--
jpam will definitely cause memory corruption. However, the version of JPAM on
public repos will also cause corruption unless a single patch has been applied.
There is a community patch for this issue that has been validated by several
sources: https://github.com/kohsuke/libpam4j/issues/16. However, the project
author has not responded to merge into another release
With the patch, libpam4j has been robust in our tests. Note that without it
corruption seems more consistent on certain Redhat versions as well as certain
newer refs of JNA (a dependency)
> Replace JPAM with libpam4j for PAM authentication
> -
>
> Key: HIVE-16529
> URL: https://issues.apache.org/jira/browse/HIVE-16529
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 1.2.0
>Reporter: Richard Ding
>Assignee: Sailaja Navvluru
>
> PAM authentication is an important feature available since Hive 0.13. But
> Hive blog gives the following warnings:
> {quote}
> JPAM library that is used to provide the PAM authentication mode can cause
> HiveServer2 to go down if a user's password has expired. This happens because
> of segfault/core dumps from native code invoked by JPAM. Some users have also
> reported crashes during logins in other cases as well. Use of LDAP or
> KERBEROS is recommended.
> {quote}
> JPAM also requires user to install a native library. Furthermore, JPAM
> library seems not to have been updated since 2007.
> Other Apache projects (e.g. Ambari/Ranger/Knox) use a newer library libpam4j
> which doesn't require installation of native library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116982#comment-16116982 ] Sergey Shelukhin commented on HIVE-17126: - Can you get the problematic statement from MariaDB logs? It would most likely need a DataNucleus bug filed. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters
[
https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116963#comment-16116963
]
Misha Dmitriev commented on HIVE-17237:
---
This is to save memory and improve performance. String.intern() has always been
an "official" solution to the string duplication problem. However, until JDK 7
it was not very scalable. This forced people to start using their own interners
based on WeakHashMap or ConcurrentHashMap. But, as we know, these data
structures are not economical at all in terms of memory - there is an overhead
of 32 bytes or more per an interned string. Starting from JDK 7, Sun/Oracle
finally paid attention and made several improvements to String.intern(), that
greatly improved its performance. The internal hashtable used by
String.intern() is also much more economical in terms of memory, and
preallocated. So since JDK 7, it became counterproductive to use custom string
interners.
> HMS wastes 26.4% of memory due to dup strings in
> metastore.api.Partition.parameters
> ---
>
> Key: HIVE-17237
> URL: https://issues.apache.org/jira/browse/HIVE-17237
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HIVE-17237.01.patch
>
>
> I've analyzed a heap dump from a production Hive installation using jxray
> (www.jxray.com) It turns out that there are a lot of duplicate strings in
> memory, that waste 26.4% of the heap. Most of them come from HashMaps
> referenced by org.apache.hadoop.hive.metastore.api.Partition.parameters.
> Below is the relevant section of the jxray report.
> Looking at Partition.java, I see that in the past somebody has already added
> code to intern keys and values in the parameters table when it's first set
> up. However, when more key-value pairs are added, they are not interned, and
> that probably explains the reason for all these duplicate strings. Also when
> a Partition instance is deserialized, no interning of parameters is currently
> done.
> {code}
> 6. DUPLICATE STRINGS
> Total strings: 3,273,557 Unique strings: 460,390 Duplicate values: 110,232
> Overhead: 3,220,458K (26.4%)
>
> ===
> 7. REFERENCE CHAINS FOR DUPLICATE STRINGS
> 2,326,150K (19.1%), 597058 dup strings (36386 unique), 597058 dup backing
> arrays:
> 39949 of "-1", 39088 of "true", 28959 of "8", 20987 of "1", 18437 of "10",
> 9583 of "9", 5908 of "269664", 5691 of "174528", 4598 of "133980", 4598 of
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length
> 3560]"
> ... and 419200 more strings, of which 36376 are unique
> Also contains one-char strings: 217 of "6", 147 of "7", 91 of "4", 28 of "5",
> 28 of "2", 21 of "0"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <--
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
> <-- Java Local
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
> [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
> 463,060K (3.8%), 119644 dup strings (34075 unique), 119644 dup backing
> arrays:
> 7914 of "true", 7912 of "-1", 6578 of "8", 5606 of "1", 2302 of "10", 1626 of
> "174528", 1223 of "9", 970 of "171680", 837 of "269664", 657 of "133980"
> ... and 84009 more strings, of which 34065 are unique
> Also contains one-char strings: 42 of "7", 31 of "6", 20 of "4", 8 of "5", 5
> of "2", 3 of "0"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.TreeMap}.values <-- Java Local (j.u.TreeMap) [@6f084afa0,@73aac9e68]
> 233,384K (1.9%), 64601 dup strings (27295 unique), 64601 dup backing arrays:
> 4472 of "true", 4173 of "-1", 3798 of "1", 3591 of "8", 813 of "174528", 684
> of "10" ... and 44568 more strings, of which 27285 are unique
> Also contains one-char strings: 305 of "7", 301 of "0", 277 of "4", 146 of
> "6", 29 of "2", 23 of "5", 19 of "9", 2 of "3"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <-- Java Local (j.u.ArrayList)
> [@4f4cfbd10,@536122408,@726616778]
> ...
> 52,916K (0.4%), 597058 dup strings (16 unique), 597058 dup backing arrays:
> <-- {j.u.HashMap}.keys <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <--
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
> <-- Java Local
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
> [@6e33618d8,@6eedb9a80,@6eedbad68,@6ee
[jira] [Updated] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[
https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17181:
Affects Version/s: 3.0.0
Status: Patch Available (was: Open)
> HCatOutputFormat should expose complete output-schema (including
> partition-keys) for dynamic-partitioning MR jobs
> -
>
> Key: HIVE-17181
> URL: https://issues.apache.org/jira/browse/HIVE-17181
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch,
> HIVE-17181.3.patch, HIVE-17181.branch-2.patch
>
>
> Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic
> partitioning are expected to call the following API methods:
> # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to
> write to. This call populates the {{OutputJobInfo}} with details fetched from
> the Metastore.
> # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data
> being written.
> It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows:
> {code:java}
> HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf));
> {code}
> Unfortunately, {{getTableSchema()}} returns only the record-schema, not the
> entire table's schema. We'll need a better API for use in M/R jobs to get the
> complete table-schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[
https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17181:
Status: Open (was: Patch Available)
> HCatOutputFormat should expose complete output-schema (including
> partition-keys) for dynamic-partitioning MR jobs
> -
>
> Key: HIVE-17181
> URL: https://issues.apache.org/jira/browse/HIVE-17181
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch,
> HIVE-17181.3.patch, HIVE-17181.branch-2.patch
>
>
> Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic
> partitioning are expected to call the following API methods:
> # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to
> write to. This call populates the {{OutputJobInfo}} with details fetched from
> the Metastore.
> # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data
> being written.
> It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows:
> {code:java}
> HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf));
> {code}
> Unfortunately, {{getTableSchema()}} returns only the record-schema, not the
> entire table's schema. We'll need a better API for use in M/R jobs to get the
> complete table-schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[
https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17181:
Attachment: HIVE-17181.3.patch
Yikes, that's a good point, [~thejas]. Here's the corrected patch.
> HCatOutputFormat should expose complete output-schema (including
> partition-keys) for dynamic-partitioning MR jobs
> -
>
> Key: HIVE-17181
> URL: https://issues.apache.org/jira/browse/HIVE-17181
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch,
> HIVE-17181.3.patch, HIVE-17181.branch-2.patch
>
>
> Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic
> partitioning are expected to call the following API methods:
> # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to
> write to. This call populates the {{OutputJobInfo}} with details fetched from
> the Metastore.
> # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data
> being written.
> It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows:
> {code:java}
> HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf));
> {code}
> Unfortunately, {{getTableSchema()}} returns only the record-schema, not the
> entire table's schema. We'll need a better API for use in M/R jobs to get the
> complete table-schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17237) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters
[
https://issues.apache.org/jira/browse/HIVE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116948#comment-16116948
]
Sahil Takiar commented on HIVE-17237:
-
Why replace the use of {{Interners.newWeakInterner}} with {{String#intern()}}?
> HMS wastes 26.4% of memory due to dup strings in
> metastore.api.Partition.parameters
> ---
>
> Key: HIVE-17237
> URL: https://issues.apache.org/jira/browse/HIVE-17237
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HIVE-17237.01.patch
>
>
> I've analyzed a heap dump from a production Hive installation using jxray
> (www.jxray.com) It turns out that there are a lot of duplicate strings in
> memory, that waste 26.4% of the heap. Most of them come from HashMaps
> referenced by org.apache.hadoop.hive.metastore.api.Partition.parameters.
> Below is the relevant section of the jxray report.
> Looking at Partition.java, I see that in the past somebody has already added
> code to intern keys and values in the parameters table when it's first set
> up. However, when more key-value pairs are added, they are not interned, and
> that probably explains the reason for all these duplicate strings. Also when
> a Partition instance is deserialized, no interning of parameters is currently
> done.
> {code}
> 6. DUPLICATE STRINGS
> Total strings: 3,273,557 Unique strings: 460,390 Duplicate values: 110,232
> Overhead: 3,220,458K (26.4%)
>
> ===
> 7. REFERENCE CHAINS FOR DUPLICATE STRINGS
> 2,326,150K (19.1%), 597058 dup strings (36386 unique), 597058 dup backing
> arrays:
> 39949 of "-1", 39088 of "true", 28959 of "8", 20987 of "1", 18437 of "10",
> 9583 of "9", 5908 of "269664", 5691 of "174528", 4598 of "133980", 4598 of
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length
> 3560]"
> ... and 419200 more strings, of which 36376 are unique
> Also contains one-char strings: 217 of "6", 147 of "7", 91 of "4", 28 of "5",
> 28 of "2", 21 of "0"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <--
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
> <-- Java Local
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
> [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
> 463,060K (3.8%), 119644 dup strings (34075 unique), 119644 dup backing
> arrays:
> 7914 of "true", 7912 of "-1", 6578 of "8", 5606 of "1", 2302 of "10", 1626 of
> "174528", 1223 of "9", 970 of "171680", 837 of "269664", 657 of "133980"
> ... and 84009 more strings, of which 34065 are unique
> Also contains one-char strings: 42 of "7", 31 of "6", 20 of "4", 8 of "5", 5
> of "2", 3 of "0"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.TreeMap}.values <-- Java Local (j.u.TreeMap) [@6f084afa0,@73aac9e68]
> 233,384K (1.9%), 64601 dup strings (27295 unique), 64601 dup backing arrays:
> 4472 of "true", 4173 of "-1", 3798 of "1", 3591 of "8", 813 of "174528", 684
> of "10" ... and 44568 more strings, of which 27285 are unique
> Also contains one-char strings: 305 of "7", 301 of "0", 277 of "4", 146 of
> "6", 29 of "2", 23 of "5", 19 of "9", 2 of "3"
> <-- {j.u.HashMap}.values <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <-- Java Local (j.u.ArrayList)
> [@4f4cfbd10,@536122408,@726616778]
> ...
> 52,916K (0.4%), 597058 dup strings (16 unique), 597058 dup backing arrays:
> <-- {j.u.HashMap}.keys <--
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--
> {j.u.ArrayList} <--
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
> <-- Java Local
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
> [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17246) Add having related blobstore query test
[ https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116944#comment-16116944 ] Taklon Stephen Wu commented on HIVE-17246: -- Great! and Thanks [~spena]] > Add having related blobstore query test > --- > > Key: HIVE-17246 > URL: https://issues.apache.org/jira/browse/HIVE-17246 > Project: Hive > Issue Type: Test >Affects Versions: 2.1.1 >Reporter: Taklon Stephen Wu >Assignee: Taklon Stephen Wu > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17246.patch > > > This patch introduces the following regression test into the hive-blobstore > qtest module: > * having.q -> Test Having clause for aggregation functions such as COUNT(), > MAX(), and min() > ** test HAVING with aggregation as alias > ** test HAVING with aggregation as field > ** test HAVING with aggregation as a function does not exist in field -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17246) Add having related blobstore query test
[ https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116944#comment-16116944 ] Taklon Stephen Wu edited comment on HIVE-17246 at 8/7/17 5:57 PM: -- Great! and Thanks [~spena] was (Author: wutak...@amazon.com): Great! and Thanks [~spena]] > Add having related blobstore query test > --- > > Key: HIVE-17246 > URL: https://issues.apache.org/jira/browse/HIVE-17246 > Project: Hive > Issue Type: Test >Affects Versions: 2.1.1 >Reporter: Taklon Stephen Wu >Assignee: Taklon Stephen Wu > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17246.patch > > > This patch introduces the following regression test into the hive-blobstore > qtest module: > * having.q -> Test Having clause for aggregation functions such as COUNT(), > MAX(), and min() > ** test HAVING with aggregation as alias > ** test HAVING with aggregation as field > ** test HAVING with aggregation as a function does not exist in field -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116933#comment-16116933
]
Xuefu Zhang commented on HIVE-17257:
Patch looks simple and good to me. Is it possible to have a test case on this?
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17213) HoS: file merging doesn't work for union all
[ https://issues.apache.org/jira/browse/HIVE-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116929#comment-16116929 ] Sahil Takiar commented on HIVE-17213: - [~csun] done. Next run of Hive QA should pick up the changes. > HoS: file merging doesn't work for union all > > > Key: HIVE-17213 > URL: https://issues.apache.org/jira/browse/HIVE-17213 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Chao Sun >Assignee: Chao Sun > Fix For: 3.0.0 > > Attachments: HIVE-17213.0.patch, HIVE-17213.1.patch, > HIVE-17213.2.patch, HIVE-17213.3.patch, HIVE-17213.4.patch, HIVE-17213.5.patch > > > HoS file merging doesn't work properly since it doesn't set linked file sinks > properly which is used to generate move tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17191) Add InterfaceAudience and InterfaceStability annotations for StorageHandler APIs
[ https://issues.apache.org/jira/browse/HIVE-17191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116926#comment-16116926 ] Aihua Xu commented on HIVE-17191: - +1. > Add InterfaceAudience and InterfaceStability annotations for StorageHandler > APIs > > > Key: HIVE-17191 > URL: https://issues.apache.org/jira/browse/HIVE-17191 > Project: Hive > Issue Type: Sub-task > Components: StorageHandler >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17191.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16758) Better Select Number of Replications
[
https://issues.apache.org/jira/browse/HIVE-16758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116848#comment-16116848
]
BELUGA BEHR commented on HIVE-16758:
[~csun] I don't have the capabilities to compile and install from scratch.
I've been relying on unit tests. Regardless of my inconvenience, it will be
much better to base this replication decision on
{{mapreduce.client.submit.file.replication}} for reasons explained in one of my
earlier comments.
Thank you for all of your time and assistance!
> Better Select Number of Replications
>
>
> Key: HIVE-16758
> URL: https://issues.apache.org/jira/browse/HIVE-16758
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-16758.1.patch, HIVE-16758.2.patch,
> HIVE-16758.3.patch
>
>
> {{org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.java}}
> We should be smarter about how we pick a replication number. We should add a
> new configuration equivalent to {{mapreduce.client.submit.file.replication}}.
> This value should be around the square root of the number of nodes and not
> hard-coded in the code.
> {code}
> public static final String DFS_REPLICATION_MAX = "dfs.replication.max";
> private int minReplication = 10;
> @Override
> protected void initializeOp(Configuration hconf) throws HiveException {
> ...
> int dfsMaxReplication = hconf.getInt(DFS_REPLICATION_MAX, minReplication);
> // minReplication value should not cross the value of dfs.replication.max
> minReplication = Math.min(minReplication, dfsMaxReplication);
> }
> {code}
> https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17213) HoS: file merging doesn't work for union all
[ https://issues.apache.org/jira/browse/HIVE-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-17213: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master branch. Thanks Xuefu for the review. [~stakiar]: can you proceed to make the ptest config change? Thanks. > HoS: file merging doesn't work for union all > > > Key: HIVE-17213 > URL: https://issues.apache.org/jira/browse/HIVE-17213 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Chao Sun >Assignee: Chao Sun > Fix For: 3.0.0 > > Attachments: HIVE-17213.0.patch, HIVE-17213.1.patch, > HIVE-17213.2.patch, HIVE-17213.3.patch, HIVE-17213.4.patch, HIVE-17213.5.patch > > > HoS file merging doesn't work properly since it doesn't set linked file sinks > properly which is used to generate move tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16758) Better Select Number of Replications
[
https://issues.apache.org/jira/browse/HIVE-16758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116819#comment-16116819
]
Chao Sun commented on HIVE-16758:
-
[~belugabehr] Now I'm looking at the Hadoop code, it is using 10 as the default
value so yes I agree we should follow that pattern.
Can you double check whether the default 10 will cause HiveOnSpark to fail on a
cluster with size less than 10?
Since we are using distributed cache for the small tables, I think it's OK to
use config. In Hadoop code it is also used for
distributed cache, job files, aux jars, etc.
> Better Select Number of Replications
>
>
> Key: HIVE-16758
> URL: https://issues.apache.org/jira/browse/HIVE-16758
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-16758.1.patch, HIVE-16758.2.patch,
> HIVE-16758.3.patch
>
>
> {{org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.java}}
> We should be smarter about how we pick a replication number. We should add a
> new configuration equivalent to {{mapreduce.client.submit.file.replication}}.
> This value should be around the square root of the number of nodes and not
> hard-coded in the code.
> {code}
> public static final String DFS_REPLICATION_MAX = "dfs.replication.max";
> private int minReplication = 10;
> @Override
> protected void initializeOp(Configuration hconf) throws HiveException {
> ...
> int dfsMaxReplication = hconf.getInt(DFS_REPLICATION_MAX, minReplication);
> // minReplication value should not cross the value of dfs.replication.max
> minReplication = Math.min(minReplication, dfsMaxReplication);
> }
> {code}
> https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17192) Add InterfaceAudience and InterfaceStability annotations for Stats Collection APIs
[ https://issues.apache.org/jira/browse/HIVE-17192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116794#comment-16116794 ] Sahil Takiar commented on HIVE-17192: - [~aihuaxu] could you take a look? RB - https://reviews.apache.org/r/61471/ > Add InterfaceAudience and InterfaceStability annotations for Stats Collection > APIs > -- > > Key: HIVE-17192 > URL: https://issues.apache.org/jira/browse/HIVE-17192 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17192.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17191) Add InterfaceAudience and InterfaceStability annotations for StorageHandler APIs
[ https://issues.apache.org/jira/browse/HIVE-17191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116789#comment-16116789 ] Sahil Takiar commented on HIVE-17191: - [~aihuaxu] could you take a look? RB - https://reviews.apache.org/r/61226/ > Add InterfaceAudience and InterfaceStability annotations for StorageHandler > APIs > > > Key: HIVE-17191 > URL: https://issues.apache.org/jira/browse/HIVE-17191 > Project: Hive > Issue Type: Sub-task > Components: StorageHandler >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17191.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17132) Add InterfaceAudience and InterfaceStability annotations for UDF APIs
[ https://issues.apache.org/jira/browse/HIVE-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116788#comment-16116788 ] Sahil Takiar commented on HIVE-17132: - [~ashutoshc] any other comments? > Add InterfaceAudience and InterfaceStability annotations for UDF APIs > - > > Key: HIVE-17132 > URL: https://issues.apache.org/jira/browse/HIVE-17132 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17132.1.patch > > > Add InterfaceAudience and InterfaceStability annotations for UDF APIs. UDFs > are a useful plugin point for Hive users, and there are a number of external > UDF libraries, such as hivemall. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17263) Reduce debug logging for S3 tables
[ https://issues.apache.org/jira/browse/HIVE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116783#comment-16116783 ] Sahil Takiar commented on HIVE-17263: - +1 I've experienced the same issues when working with S3. Whenever debug logging is enabled (e.g. during ptest runs), the S3 client dumps a lot of logs, most of which are not useful. > Reduce debug logging for S3 tables > -- > > Key: HIVE-17263 > URL: https://issues.apache.org/jira/browse/HIVE-17263 > Project: Hive > Issue Type: Improvement >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17263.01.patch > > > When log level is set to debug operations accessing tables on amazon s3 will > output a significant amount of logs, a lot of which is about the http > communication (http headers and requests) which may not be that useful even > for debugging purposes. > Since some ZooKeeper, Hadoop, DataNucleus etc. loggers are by default set to > INFO+ levels I suggest we do the same for Apache Http and AWS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16896) move replication load related work in semantic analysis phase to execution phase using a task
[
https://issues.apache.org/jira/browse/HIVE-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116763#comment-16116763
]
Sankar Hariappan commented on HIVE-16896:
-
+1.
The 3.patch looks good to me.
> move replication load related work in semantic analysis phase to execution
> phase using a task
> -
>
> Key: HIVE-16896
> URL: https://issues.apache.org/jira/browse/HIVE-16896
> Project: Hive
> Issue Type: Sub-task
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16896.1.patch, HIVE-16896.2.patch,
> HIVE-16896.3.patch
>
>
> we want to not create too many tasks in memory in the analysis phase while
> loading data. Currently we load all the files in the bootstrap dump location
> as {{FileStatus[]}} and then iterate over it to load objects, we should
> rather move to
> {code}
> org.apache.hadoop.fs.RemoteIteratorlistFiles(Path
> f, boolean recursive)
> {code}
> which would internally batch and return values.
> additionally since we cant hand off partial tasks from analysis pahse =>
> execution phase, we are going to move the whole repl load functionality to
> execution phase so we can better control creation/execution of tasks (not
> related to hive {{Task}}, we may get rid of ReplCopyTask)
> Additional consideration to take into account at the end of this jira is to
> see if we want to specifically do a multi threaded load of bootstrap dump.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17160) Adding kerberos Authorization to the Druid hive integration
[ https://issues.apache.org/jira/browse/HIVE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17160: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! > Adding kerberos Authorization to the Druid hive integration > --- > > Key: HIVE-17160 > URL: https://issues.apache.org/jira/browse/HIVE-17160 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra > Fix For: 3.0.0 > > Attachments: HIVE-17160.2.patch, HIVE-17160.patch > > > This goal of this feature is to allow hive querying a secured druid cluster > using kerberos credentials. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17263) Reduce debug logging for S3 tables
[
https://issues.apache.org/jira/browse/HIVE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116737#comment-16116737
]
Hive QA commented on HIVE-17263:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880649/HIVE-17263.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10992 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=239)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=239)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6280/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6280/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880649 - PreCommit-HIVE-Build
> Reduce debug logging for S3 tables
> --
>
> Key: HIVE-17263
> URL: https://issues.apache.org/jira/browse/HIVE-17263
> Project: Hive
> Issue Type: Improvement
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17263.01.patch
>
>
> When log level is set to debug operations accessing tables on amazon s3 will
> output a significant amount of logs, a lot of which is about the http
> communication (http headers and requests) which may not be that useful even
> for debugging purposes.
> Since some ZooKeeper, Hadoop, DataNucleus etc. loggers are by default set to
> INFO+ levels I suggest we do the same for Apache Http and AWS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17263) Reduce debug logging for S3 tables
[ https://issues.apache.org/jira/browse/HIVE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116727#comment-16116727 ] Peter Vary commented on HIVE-17263: --- I think it would be useful to reduce the noise, but since I have not been working too much with S3, I would appreciate your view on this [~spena], [~stakiar], [~vihangk1], [~kuczoram]. Thanks, Peter > Reduce debug logging for S3 tables > -- > > Key: HIVE-17263 > URL: https://issues.apache.org/jira/browse/HIVE-17263 > Project: Hive > Issue Type: Improvement >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17263.01.patch > > > When log level is set to debug operations accessing tables on amazon s3 will > output a significant amount of logs, a lot of which is about the http > communication (http headers and requests) which may not be that useful even > for debugging purposes. > Since some ZooKeeper, Hadoop, DataNucleus etc. loggers are by default set to > INFO+ levels I suggest we do the same for Apache Http and AWS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15176) Small typo in hiveserver2 webui
[ https://issues.apache.org/jira/browse/HIVE-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-15176: -- Resolution: Duplicate Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Resolved by another commit: HIVE-17067 > Small typo in hiveserver2 webui > --- > > Key: HIVE-15176 > URL: https://issues.apache.org/jira/browse/HIVE-15176 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Web UI >Reporter: Miklos Csanady >Assignee: Miklos Csanady >Priority: Trivial > Labels: trivial > Fix For: 3.0.0 > > Attachments: HIVE-15176.patch > > > I found a small typo in webui for hiveserver2 > The Waited is happened to spell Wtaited. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17263) Reduce debug logging for S3 tables
[ https://issues.apache.org/jira/browse/HIVE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17263: --- Attachment: HIVE-17263.01.patch > Reduce debug logging for S3 tables > -- > > Key: HIVE-17263 > URL: https://issues.apache.org/jira/browse/HIVE-17263 > Project: Hive > Issue Type: Improvement >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17263.01.patch > > > When log level is set to debug operations accessing tables on amazon s3 will > output a significant amount of logs, a lot of which is about the http > communication (http headers and requests) which may not be that useful even > for debugging purposes. > Since some ZooKeeper, Hadoop, DataNucleus etc. loggers are by default set to > INFO+ levels I suggest we do the same for Apache Http and AWS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
