[jira] [Created] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18323:
---

 Summary: Vectorization: add the support of timestamp in 
VectorizedPrimitiveColumnReader
 Key: HIVE-18323
 URL: https://issues.apache.org/jira/browse/HIVE-18323
 Project: Hive
  Issue Type: Improvement
  Components: Vectorization
Affects Versions: 3.0.0
Reporter: Aihua Xu


{noformat}
CREATE TABLE `t1`(
  `ts` timestamp,
  `s1` string)
STORED AS PARQUET;

set hive.vectorized.execution.enabled=true;
SELECT * from t1 SORT BY s1;
{noformat}

This query will throw exception since timestamp is not supported here yet.

{noformat}
Caused by: java.io.IOException: java.io.IOException: Unsupported type: optional 
int96 ts
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
at 
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18322) RetryingMetaStoreClient reconnect should not use ugi.doAs if not necessary

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-18322:


 Summary: RetryingMetaStoreClient reconnect should not use ugi.doAs 
if not necessary
 Key: HIVE-18322
 URL: https://issues.apache.org/jira/browse/HIVE-18322
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair


As commented in HIVE-17853 , RetryingMetaStoreClient should also check to see 
if current user is same as the original UGI user, and not do the ugi.doAs() if 
it is the same. Otherwise, this can potentially cause problems where the users 
are not privileged users (ie, there is no intent to do a "doAs").
Without such a check, you would get errors like " userX is not allowed to 
impersonate userX".




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18321) Support REBUILD for MVs backed by custom storage handlers

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18321:
--

 Summary: Support REBUILD for MVs backed by custom storage handlers
 Key: HIVE-18321
 URL: https://issues.apache.org/jira/browse/HIVE-18321
 Project: Hive
  Issue Type: Bug
  Components: Druid integration, Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


For instance, currently REBUILD will fail for Druid backed MVs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18320) Support ACID Tables replication

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-18320:
---

 Summary: Support ACID Tables replication
 Key: HIVE-18320
 URL: https://issues.apache.org/jira/browse/HIVE-18320
 Project: Hive
  Issue Type: New Feature
  Components: HiveServer2, Metastore, repl, Transactions
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Currently, Full ACID and MM (Micro-Managed) tables are not supported by Replv2.
Need to support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Adding Hive Metastore functions to add and alter partitions for multiple tables

[秦凯捷]

Thank you all for the help. I'm preparing the patch for reviewing.

秦凯捷
Tel: +86-13810485829
E-mail: daniel...@gmail.com



On Tue, Dec 19, 2017 at 12:49 AM, Eugene Koifman 
wrote:

> +1 to Alex’ comment
>
> On 12/14/17, 3:27 PM, "Alexander Kolbasov"  wrote:
>
> Kaijie,
>
> can you describe in more details why would you need such functionality?
> What problem does it actually solve?
>
> I do not think that HMS should do more "atomic" compound operations
> then it
> does now - IMO it should do less instead. This is especially the case
> when
> operations involve a mix of metadata operations and filesystem
> operations
> which can not be always reverted correctly. Such things make semantics
> of
> HMS calls more and more complex and difficult to maintain. Existing
> bulk
> APIs are not a good example that we should follow.
>
>
> - Alex
>
> On Wed, Dec 13, 2017 at 6:54 PM, 秦凯捷  wrote:
>
> > Hi Andrew,
> >
> > Thanks for you response. For your comments:
> >
> > -Functionality:
> > Support adding and altering multiple partitions for multiple tables
> in one
> > SQL and API request as one transaction.
> >
> > - what happens in the case of a failure when part way through the
> > operations.
> > For altering and adding partitions, all the objectstore changes for
> > partitions will be operated in one transaction. So the transaction
> will be
> > roll-back in case of failure.
> > For adding partitions, there may be additional steps to add
> directories on
> > filesystem for newly added partitions. They will be deleted in case
> of
> > failure, just like what AddPartitions is doing now.
> >
> > - what impact on the system there will be if an operation takes a
> long time
> > Alter partitions for multiple tables actually has no big difference
> than
> > current altering partitions for one table. They will both take a
> long time
> > if someone is trying to alter too many partitions or for too many
> tables.
> > Transaction timeout will strike down the operation.
> > We are doing performance test on our system to see how long it takes
> for
> > multiple scenarios but after all, this should not be a blocker.
> >
> > Thanks,
> > Kaijie
> >
> > 秦凯捷
> > Tel: +86-13810485829
> > E-mail: daniel...@gmail.com
> >
> >
> >
> > On Thu, Dec 14, 2017 at 3:38 AM, Andrew Sherman <
> asher...@cloudera.com>
> > wrote:
> >
> > > Hi Kaijie,
> > >
> > > I think this is an area that other the Hive community is
> interested in.
> > So
> > > please do go ahead and describe your functionality.
> > > I think that it is important to describe
> > > - what happens in the case of a failure when part way through the
> > > operations.
> > > - what impact on the system there will be if an operation takes a
> long
> > time
> > >
> > > Thanks
> > >
> > > -Andrew
> > >
> > > On Tue, Dec 12, 2017 at 1:31 AM, 秦凯捷  wrote:
> > >
> > > > Hi dev,
> > > >
> > > > I'm wondering if Hive community have ever considered support
> adding and
> > > > altering multiple partitions for multiple tables?
> > > >
> > > > I'm using Hive Metastore to manage the metadata for Presto
> querying.
> > Our
> > > > business requires that we should publish some partitions of data
> for
> > > > multiple tables at the same time in an atomic transaction to
> keep the
> > > data
> > > > consistency. Currently Hive Metastore only supports adding and
> altering
> > > > multiple tables for one table.
> > > >
> > > > I drafted AddPartitionsForTables and AlterPartitionsForTables
> function
> > to
> > > > achieve this based on existing AddPartition and AlterPartition
> logic
> > and
> > > we
> > > > are testing it on our system.
> > > > I'm wondering if community have considered these functionality.
> I would
> > > > like to contribute the functionality if you have interest.
> > > >
> > > > Thank you!
> > > > -Kaijie
> > > >
> > > >
> > > > Tel: +86-13810485829
> > > > E-mail: daniel...@gmail.com
> > > >
> > >
> >
>
>
>

[GitHub] hive pull request #283: HIVE-17829: ArrayIndexOutOfBoundsException - HBASE-b...

[anishek]

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/283

HIVE-17829: ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro 
schema in Hive2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-17829

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/283.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #283


commit 5c0b586af9323a2095c24507665064a32252834b
Author: Anishek Agarwal 
Date:   2017-12-20T08:57:43Z

HIVE-17829: ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro 
schema in Hive2




---

[GitHub] hive pull request #282: HIVE-18290: hbase backed table creation fails where ...

[anishek]

Github user anishek closed the pull request at:

https://github.com/apache/hive/pull/282


---

[GitHub] hive pull request #282: HIVE-18290: hbase backed table creation fails where ...

[anishek]

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/282

HIVE-18290: hbase backed table creation fails where no column comments 
present



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-17829

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/282.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #282


commit 1a544e7fe577ff4862638e581d0660a3169677d9
Author: Anishek Agarwal 
Date:   2017-12-18T10:27:27Z

HIVE-18290: hbase backed table creation fails where no column comments 
present




---

[GitHub] hive pull request #281: HIVE-18290: hbase backed table creation fails where ...

[anishek]

Github user anishek closed the pull request at:

https://github.com/apache/hive/pull/281


---