Hive generating different DAGs from the same query

2018-07-11 Thread Sungwoo Park
Hello, I am running the TPC-DS benchmark using Hive 3.0, and I find that Hive sometimes produces different DAGs from the same query. These are the two scenarios for the experiment. The execution engine is tez, and the TPC-DS scale factor is 3TB. 1. Run query 19 to query 24 sequentially in the sam

Re: Alter table using Thrift client

2018-07-11 Thread Peter Vary
Hi Sylvester, You can set this specific configuration value per session since HIVE-17832 - Fixed in: Hive 3.0.0, Hive 2.4.0 So you can do this change this value through thrift if your metastore version is higher or equal than 2.4.0, or 3.0.0, but not with 1.2.1 If you want to use 1.2.1 version

Standalone metastore

2018-07-11 Thread Sandhya Agarwal
Hello, We want to leverage standalone metastore for our project, for which I want to enable access to multiple S3 buckets, each with its own access key and secret key. I am trying to access the metastore operations from a Java client using HiveMetastoreClient and using the thrift metastore URI to

Re: Alter table using Thrift client

2018-07-11 Thread Sylvester Steele
Hi Peter, I upgraded the metastore jar to 3.0.0 and when I do: ThriftHiveMetastore.Client client = new ThriftHiveMetastore.Client(protocol); client.setMetaConf("hive.metastore.disallow.incompatible.col.type.changes", "false"); I get: MetaException(message:Invalid configuration key hive.metastor

Re: Alter table using Thrift client

2018-07-11 Thread Peter Vary
Hi Sylvester, Probably the deprecated configurations are not handled here. Could you please try metastore.disallow.incompatible.col.type.changes? Thanks, Peter > On Jul 11, 2018, at 15:20, Sylvester Steele > wrote: > > Hi Peter, > > I upgraded the metastore jar to 3.0.0 and when I do: > > T

Re: Alter table using Thrift client

2018-07-11 Thread Sylvester Steele
Hi Peter, client.setMetaConf("metastore.disallow.incompatible.col.type.changes", "false") gets me the same error: MetaException(message:Invalid configuration key metastore.disallow.incompatible.col.type.changes) Thanks On Wed, Jul 11, 2018 at 12:56 PM, Peter Vary wrote: > Hi Sylvester, > > Pr

Re: Standalone metastore

2018-07-11 Thread Vihang Karajgaonkar
AFAIK currently, s3 keys cannot be session specific currently. They are loaded by the metastore server when it starts and it cannot be modified without a metastore server restart. In order to do this we will have make some code changes. I had created HIVE-16913 for this long time back but never got

Re: Standalone metastore

2018-07-11 Thread Sandhya Agarwal
Thank you for the revert. We do have a use case to provide tenant / bucket specific keys for a tenant user session, without having to restart the MetaStore server when a new tenant is provisioned. Nevertheless, let me explore a bit around Sentry / Ranger too. Thank you, Sandhya On Thu, Jul 12,