[jira] [Created] (HIVE-18874) JDBC: HiveConnection expects custom logging setup

[Gopal V (JIRA)]

Gopal V created HIVE-18874:
--

 Summary: JDBC: HiveConnection expects custom logging setup
 Key: HIVE-18874
 URL: https://issues.apache.org/jira/browse/HIVE-18874
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


This prevents Hive JDBC from being instantiated into a regular SLF4J logger env.

{code}
java.lang.IncompatibleClassChangeError: Class 
org.apache.logging.slf4j.Log4jLoggerFactory does not implement the requested 
interface org.apache.hive.org.slf4j.ILoggerFactory
at 
org.apache.hive.org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18875) Enable SMB Join by default in Tez

[Deepak Jaiswal (JIRA)]

Deepak Jaiswal created HIVE-18875:
-

 Summary: Enable SMB Join by default in Tez
 Key: HIVE-18875
 URL: https://issues.apache.org/jira/browse/HIVE-18875
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal
 Attachments: HIVE-18875.1.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65415: HIVE-18571 stats issues for MM tables

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65415/
---

(Updated March 6, 2018, 8:44 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

f.,v fbghdscd


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java df77a4a2f2 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d955b48360 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
 41c89b1cd3 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
 6e8d6b62a5 
  ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 9ea7a7c79b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java eee5e66ea7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java b490325091 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 804cd7868b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 0a82225d4a 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java ced84b3e15 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java 1a63d3f971 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c0be51e0b2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
1c6b793e11 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
7d2de75315 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7f446ca1de 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java a4e770ce95 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
8ce0cb05b6 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
946c300750 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 1d7660e8b2 
  ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java 7591c0681b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 05b0474e90 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java 
d84cf136d5 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java 
922cfc23c0 
  ql/src/test/queries/clientnegative/orc_change_fileformat_acid.q cc73616a32 
  ql/src/test/queries/clientnegative/orc_change_serde_acid.q 91a2be50c0 
  ql/src/test/queries/clientnegative/orc_reorder_columns1_acid.q 234e74bb74 
  ql/src/test/queries/clientnegative/orc_reorder_columns2_acid.q 57ab049c6d 
  ql/src/test/queries/clientnegative/orc_replace_columns1_acid.q 9fe9209d03 
  ql/src/test/queries/clientnegative/orc_replace_columns2_acid.q 7b37757ebf 
  ql/src/test/queries/clientnegative/orc_replace_columns3_acid.q e3cb819b62 
  ql/src/test/queries/clientnegative/orc_type_promotion1_acid.q 3a8c08a829 
  ql/src/test/queries/clientnegative/orc_type_promotion2_acid.q 1d24b1dd18 
  ql/src/test/queries/clientnegative/orc_type_promotion3_acid.q 83764e29cc 
  ql/src/test/queries/clientpositive/acid_nullscan.q d048231584 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 669fa3fa47 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 1f4c0adfc7 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 1abd3a29f1 
  ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
64e5b17936 
  ql/src/test/results/clientpositive/llap/default_constraint.q.out 89b1224004 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
6a97736008 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
352f6bad01 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
 d8863a2c80 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_rebuild_dummy.q.out
 d8863a2c80 
  ql/src/test/results/clientpositive/llap/mm_all.q.out 23e733b4c0 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 
29e408c60c 
  ql/src/test/results/clientpositive/mm_all.q.out ac6c08057c 
  ql/src/test/results/clientpositive/mm_default.q.out 1345efdfb6 
  ql/src/test/results/clientpositive/tez/acid_vectorization_original_tez.q.out 
92a04ddbf3 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out 7f18f2b42b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
 59190893e6 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2d34 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 2be018ba0f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 20c10607bb 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java
 b44ff8ce47 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013 
  

Re: HMS API testing and direct SQL

[Alan Gates]

I would do it at the ObjectStore level.  It's faster and will cover 99% of
what we want to test between the two.

Alan.

On Tue, Mar 6, 2018 at 8:10 AM, Peter Vary  wrote:

> Hi Team,
>
> The MetaStore API tests are designed to be easily extensible by new
> MetaStore configurations. Just take a look at the following class:
> MetaStoreFactoryForTests. We can add a new configurations here like it is
> done with the RemoteMetaStore.
> What we have to decide is: which is the right level to test DirectSQL
> methods. MetaStore level or on ObjectStore level. I can see pros and cons
> about both solutions. What do you think?
>
> Thanks,
> Peter
>
> > On Mar 2, 2018, at 8:14 PM, Sergey Shelukhin 
> wrote:
> >
> > There used to be a special ObjectStore for tests that would run stuff
> both
> > with and without DirectSql in tests and fail on mismatch. Not sure about
> > its state now.
> >
> > On 18/3/2, 11:05, "Vihang Karajgaonkar"  wrote:
> >
> >> Direct SQL is controlled by configs metastore.try.direct.sql
> >> and metastore.try.direct.sql.ddl which are enabled by default. So I am
> >> guessing the API tests will cover the direct SQL enabled path only
> unless
> >> the tests override the default configs (I don't think they do).
> >>
> >> On Fri, Mar 2, 2018 at 10:51 AM, Alexander Kolbasov  >
> >> wrote:
> >>
> >>> We currently use two distinct code paths via ObjectStore - the regular
> >>> one
> >>> and direct SQL. So I think all the tests that verify HMS API
> >>> functionality
> >>> should somehow cover both cases.
> >>>
> >>> Peter - what are your thoughts?
> >>>
> >>> - Alex
> >>>
> >
>
>

Re: HMS API testing and direct SQL

[Sergey Shelukhin]

I meant VerifyingObjectStore. I still see it in tests, at least when 
investigating a random test failure yesterday.

From: Peter Vary >
Date: Tuesday, March 6, 2018 at 08:10
To: Sergey Shelukhin >
Cc: "dev@hive.apache.org" 
>
Subject: Re: HMS API testing and direct SQL

Hi Team,

The MetaStore API tests are designed to be easily extensible by new MetaStore 
configurations. Just take a look at the following class: 
MetaStoreFactoryForTests. We can add a new configurations here like it is done 
with the RemoteMetaStore.
What we have to decide is: which is the right level to test DirectSQL methods. 
MetaStore level or on ObjectStore level. I can see pros and cons about both 
solutions. What do you think?

Thanks,
Peter

On Mar 2, 2018, at 8:14 PM, Sergey Shelukhin 
> wrote:

There used to be a special ObjectStore for tests that would run stuff both
with and without DirectSql in tests and fail on mismatch. Not sure about
its state now.

On 18/3/2, 11:05, "Vihang Karajgaonkar" 
> wrote:

Direct SQL is controlled by configs metastore.try.direct.sql
and metastore.try.direct.sql.ddl which are enabled by default. So I am
guessing the API tests will cover the direct SQL enabled path only unless
the tests override the default configs (I don't think they do).

On Fri, Mar 2, 2018 at 10:51 AM, Alexander Kolbasov 
>
wrote:

We currently use two distinct code paths via ObjectStore - the regular
one
and direct SQL. So I think all the tests that verify HMS API
functionality
should somehow cover both cases.

Peter - what are your thoughts?

- Alex

[jira] [Created] (HIVE-18885) Cascaded alter table + notifications = disaster

[Alexander Kolbasov (JIRA)]

Alexander Kolbasov created HIVE-18885:
-

 Summary: Cascaded alter table + notifications = disaster
 Key: HIVE-18885
 URL: https://issues.apache.org/jira/browse/HIVE-18885
 Project: Hive
  Issue Type: Bug
  Components: Hive, Metastore
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov


You can see the problem from looking at the code, but it actually created 
severe problems for real life Hive user.

When {{alter table}} has {{cascade}} option it does the following:
{code:java}
 msdb.openTransaction()
  ...
  List parts = msdb.getPartitions(dbname, name, -1);
  for (Partition part : parts) {
List oldCols = part.getSd().getCols();
part.getSd().setCols(newt.getSd().getCols());
String oldPartName = 
Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
updatePartColumnStatsForAlterColumns(msdb, part, oldPartName, 
part.getValues(), oldCols, part);
msdb.alterPartition(dbname, name, part.getValues(), part);
  }
 {code}

So it walks all partitions (and this may be huge list) and does some 
non-trivial operations in one single uber-transaction.

When DbNotificationListener is enabled, it adds an event for each partition, 
all while
holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is 
happening no other write DDL can proceed. This can sometimes cause DB lock 
timeouts which cause HMS level operation retries which make things even worse.

In one particular case this pretty much made HMS unusable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing

[Andrew Sherman (JIRA)]

Andrew Sherman created HIVE-18877:
-

 Summary: HiveSchemaTool.validateSchemaTables() should wrap a 
SQLException when rethrowing
 Key: HIVE-18877
 URL: https://issues.apache.org/jira/browse/HIVE-18877
 Project: Hive
  Issue Type: Bug
Reporter: Andrew Sherman
Assignee: Andrew Sherman


If schematool is run with the -verbose flag then it will print a stack trace 
for an exception that occurs. If a SQLException is caught during 
HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown 
containing the text of the SQLException. If we instead throw  a 
HiveMetaException that wraps the SQLException then the stacktrace will help 
with diagnosis of issues where the SQLException contains a generic error text. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18883) Add findbugs to yetus pre-commit checks

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18883:
---

 Summary: Add findbugs to yetus pre-commit checks
 Key: HIVE-18883
 URL: https://issues.apache.org/jira/browse/HIVE-18883
 Project: Hive
  Issue Type: Sub-task
  Components: Testing Infrastructure
Reporter: Sahil Takiar
Assignee: Sahil Takiar


We should enable FindBugs for our YETUS pre-commit checks, this will help 
overall code quality and should decrease the overall number of bugs in Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18876) Remove Superfluous Logging in Driver

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18876:
--

 Summary: Remove Superfluous Logging in Driver
 Key: HIVE-18876
 URL: https://issues.apache.org/jira/browse/HIVE-18876
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/a4198f584aa0792a16d1e1eeb2ef3147403b8acb/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L2188-L2190]
{code:java}
if (console != null) {
  console.printInfo("OK");
}
{code}
 # Console can never be 'null'
 # OK is not an informative logging message, and in the HiveServer2 logs, it is 
often interwoven into the logging and is pretty much useless on its own, 
without having to trace back through the logs to see what happened before it.  
This is also printed out, even if an error occurred

Please remove this block of code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18878) Lower MoveTask Lock Logging to Debug

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18878:
--

 Summary: Lower MoveTask Lock Logging to Debug
 Key: HIVE-18878
 URL: https://issues.apache.org/jira/browse/HIVE-18878
 Project: Hive
  Issue Type: Improvement
  Components: Locking
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/cbb9233a3b39ab8489d777fc76f0758c49b69bef/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java#L233-L234

{code}
LOG.info("about to release lock for output: {} lock: {}", output,
  lock.getHiveLockObject().getName());
try {
  lockMgr.unlock(lock);
} catch (LockException le) {
{code}

Change this to _debug_ level logging.  The lock manager's 'unlock' method 
should have additional logging available.  Knowing that the unlock is about to 
take place is only helpful for debugging, not for normal operation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18881) Lower Logging for FSStatsAggregator

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18881:
--

 Summary: Lower Logging for FSStatsAggregator
 Key: HIVE-18881
 URL: https://issues.apache.org/jira/browse/HIVE-18881
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101

{code}
  LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
{code}

# All the other logging in this class is _debug_ or _error_ level logging.  
This should be debug
# Remove tab characters to allow splitting on tabs in any kind of tab-separated 
file of log lines
# Use SLF4J parameterized logging



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18887) Improve preserving column stats for alter table commands

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18887:
---

 Summary: Improve preserving column stats for alter table commands
 Key: HIVE-18887
 URL: https://issues.apache.org/jira/browse/HIVE-18887
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


We are trying to preserve column stats for certain alter table commands, while 
seems that current generic approach which compare the old columns against the 
new columns and update for all the columns may not be efficient . e.g., if we 
just rename the table, we should be able to update the name itself. COL_STATS 
table somehow contains DB_Name and Table_Name. If those tables don't have these 
columns, certain commands don't even need to update these tables. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18886) ACID: NPE on unexplained mysql exceptions

[Gopal V (JIRA)]

Gopal V created HIVE-18886:
--

 Summary: ACID: NPE on unexplained mysql exceptions 
 Key: HIVE-18886
 URL: https://issues.apache.org/jira/browse/HIVE-18886
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Gopal V


At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql 
exceptions

{code}
2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]: 
ql.Driver (:()) - FAILED: Error in acquiring locks: null
java.lang.NullPointerException
at 
org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
{code}

{code}
return e instanceof SQLTransactionRollbackException
|| ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct == 
SQLSERVER)
&& e.getSQLState().equals("40001"))
|| (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
|| (dbProduct == ORACLE && (e.getMessage().contains("deadlock detected")
|| e.getMessage().contains("can't serialize access for this 
transaction")));
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18884) Simplify Logging in Hive Metastore Client

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18884:
--

 Summary: Simplify Logging in Hive Metastore Client
 Key: HIVE-18884
 URL: https://issues.apache.org/jira/browse/HIVE-18884
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java

The current logging is:

{code}
2018-02-26 07:02:44,883  INFO  hive.metastore: [HiveServer2-Handler-Pool: 
Thread-65]: Trying to connect to metastore with URI 
thrift://host.company.com:9083
2018-02-26 07:02:44,892  INFO  hive.metastore: [HiveServer2-Handler-Pool: 
Thread-65]: Connected to metastore.
2018-02-26 07:02:44,892  INFO  hive.metastore: [HiveServer2-Handler-Pool: 
Thread-65]: Opened a connection to metastore, current connections: 2
{code}

Please simplify to something like:

{code}
2018-02-26 07:02:44,892  INFO  hive.metastore: [HiveServer2-Handler-Pool: 
Thread-65]: Opened a connection to metastore (URI 
thrift://host.company.com:9083), current connections: 2

... or ...

2018-02-26 07:02:44,892  ERROR  hive.metastore: [HiveServer2-Handler-Pool: 
Thread-65]: Failed to connect to the Metastore Sserver (URI 
thrift://host.company.com:9083)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18888) Replace synchronizedMap with ConcurrentHashMap

[Alexander Kolbasov (JIRA)]

Alexander Kolbasov created HIVE-1:
-

 Summary: Replace synchronizedMap with ConcurrentHashMap
 Key: HIVE-1
 URL: https://issues.apache.org/jira/browse/HIVE-1
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0, 2.3.3
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


There are a bunch of places that use Collections.synchronizedMap instead of 
ConcurrentHashMap which are better. We should search/replace the uses.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18879) Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-18879:
-

 Summary: Disallow embedded element in UDFXPathUtil needs to work 
if xercesImpl.jar in classpath
 Key: HIVE-18879
 URL: https://issues.apache.org/jira/browse/HIVE-18879
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18880) Change Log to Debug in CombineHiveInputFormat

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18880:
--

 Summary: Change Log to Debug in CombineHiveInputFormat
 Key: HIVE-18880
 URL: https://issues.apache.org/jira/browse/HIVE-18880
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L467]
{code:java}
  private InputSplit[] getCombineSplits(JobConf job, int numSplits,
  Map pathToPartitionInfo) {
...
  LOG.info("number of splits " + result.size());
...
}
{code}
[https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L587]
{code:java}
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
...
  LOG.info("Number of all splits " + result.size());
...
}
{code}
 # Capitalize "N"umber in the first logging to be consistent across all logging 
statements
 # Change the first logging message to be _debug_ level seeing as it's in a 
private method.
 It's an implementation logging and the entire total (most useful for a client) 
is captured in _info_ level at the end of the public method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18882) Do Not Hide Exception in Hive Metastore Client Connection

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18882:
--

 Summary: Do Not Hide Exception in Hive Metastore Client Connection
 Key: HIVE-18882
 URL: https://issues.apache.org/jira/browse/HIVE-18882
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]

 
{code:java}
if (LOG.isDebugEnabled()) {
  LOG.warn("Failed to connect to the MetaStore Server...", e);
} else {
  // Don't print full exception trace if DEBUG is not on.
  LOG.warn("Failed to connect to the MetaStore Server...");
}
{code}

I do not understand the logic here.  I always want to see the reason for the 
failure. Otherwise, I do not know why it is failing unless I restart the server 
with debug logging enabled.  By that point, the error may have cleared.  Please 
just use the Exception in the WARN output without adding confusing logging for 
debugging.  This is never an expected behavior... that enabling debug would 
change a _warn_ level log message.

Also... please remove the ellipsis, they add no value. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65943: HIVE-18888: Replace synchronizedMap with ConcurrentHashMap

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65943/
---

Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Peter Vary, 
Sahil Takiar, and Vihang Karajgaonkar.


Bugs: HIVE-1
https://issues.apache.org/jira/browse/HIVE-1


Repository: hive-git


Description
---

HIVE-1: Replace synchronizedMap with ConcurrentHashMap


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
0bed22a8f805ed64ba1af260434a40667004e051 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
c0be51e0b2c085896739a14096146822e4599d41 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 2be018ba0fd88daf484d8e20ddb64087958bed55 


Diff: https://reviews.apache.org/r/65943/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/#review198753
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2887 (original), 2889 (patched)


Here you can specify the length:

`List partitionsToAdd = new ArrayList<>(parts.size());
List partValWrappers = new ArrayList<>(parts.size());
`

But also why do we needd this list at all? We can just use partValWrappers 
as a collection of partitions we care about.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2888 (original), 2890 (patched)


You use this to check for duplicates and list is pretty bad structure for 
this - please use set instead.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2891 (original), 2897 (patched)


In cases like this it is also quite useful to know actual table and dbname 
that was supplied - it could help to figure out what wrong partition ended up 
here.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2898 (original), 2904 (patched)


Can you fix this as well to use

`LOG.info("Not adding partition {} as it already exists", part)`



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 2922 (patched)


Please use new ArrayList<>(partitionsToAdd.size())



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2902 (original), 2925 (patched)


ugi doesn't change in the loop, so it can be moved outside. Same goes for 
currentUser - it can be done just once outside the loop.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2906 (original), 2929 (patched)


Why are we converting IO exception to RuntimeException? This doesn't look 
right.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2909 (original), 2932 (patched)


I am wondering what is the motivation for doing this concurrently. I guess 
that if the list of partitions is huge, it may be useful, but for smaller lists 
it is probably just an overhead. This is putside your scope but definitely 
worth investigating.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2913 (original), 2936 (patched)


Since we are using Java 8 we can use lambdas now, so this can become:

  partFutures.add(threadPool.submit(() -> {
if (failureOccurred.get()) {
  return null;
}
ugi.doAs((PrivilegedExceptionAction) () -> {
  try {
boolean madeDir = createLocationForAddedPartition(table, 
part);
addedPartitions.put(partWrapper, madeDir);
initializeAddedPartition(table, part, madeDir);
  } catch (MetaException e) {
throw new IOException(e.getMessage(), e);
  }
  return null;
});
return part;
  }));



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2917 (original), 2941 (patched)


If we iterate over PartValEqWrapper objects, then we do not need to create 
a new one here.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2950 (original), 2984 (patched)


From the code below it looks equivalent to just

if (!newParts.isEmpty()) {
  ms.addPartitions(dbName, tblName, newParts);
}



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2957 (original), 2991 (patched)


I would suggest using different variable for this and for the success 
above. But anyway, setting of success above seems useless.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 3095 (original), 3129 (patched)


Similar comments apply to this function.


- Alexander Kolbasov


On March 

[jira] [Created] (HIVE-18890) Lower Logging for "Table not found" Error

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18890:
--

 Summary: Lower Logging for "Table not found" Error
 Key: HIVE-18890
 URL: https://issues.apache.org/jira/browse/HIVE-18890
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/7cb31c03052b815665b3231f2e513b9e65d3ff8c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1105

{code:java}
// Get the table from metastore
org.apache.hadoop.hive.metastore.api.Table tTable = null;
try {
  tTable = getMSC().getTable(dbName, tableName);
} catch (NoSuchObjectException e) {
  if (throwException) {
LOG.error("Table " + tableName + " not found: " + e.getMessage());
throw new InvalidTableException(tableName);
  }
  return null;
} catch (Exception e) {
  throw new HiveException("Unable to fetch table " + tableName + ". " + 
e.getMessage(), e);
}
{code}

We should throw an exception or log it, but not both. Right [~mdrob] ? ;)

And in this case, we are generating scary ERROR level logging in the 
HiveServer2 logs needlessly.  This should not be reported as an application 
error.  It is a simple user error, indicated by catching the 
_NoSuchObjectException_ Throwable, that can always be ignored by the service.  
It is most likely a simple user typo of the table name.  However, the more 
serious general _Exception_ is not logged.  This is backwards.

Please remove the _error_ level logging for the user error... or lower it to 
_debug_ level logging.

Please include an _error_ level logging to the general Exception case, unless 
this Exception is being captured up the stack, somewhere else, and is being 
logged there at ERROR level logging.

{code}
-- Sample log messages found in HS2 logs
2018-03-02 10:26:40,363  ERROR hive.ql.metadata.Hive: 
[HiveServer2-Handler-Pool: Thread-4467]: Table default not found: 
default.default table not found
2018-03-02 10:26:40,367  ERROR hive.ql.metadata.Hive: 
[HiveServer2-Handler-Pool: Thread-4467]: Table default not found: 
default.default table not found
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18889) update all parts of Hive to use the same Guava version

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18889:
---

 Summary: update all parts of Hive to use the same Guava version
 Key: HIVE-18889
 URL: https://issues.apache.org/jira/browse/HIVE-18889
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HIVE-18889.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18891) I am trying to write a basic query in hive and it is giving me error

[Amit Chauhan (JIRA)]

Amit Chauhan created HIVE-18891:
---

 Summary: I am trying to write a basic  query in hive and it is 
giving me error
 Key: HIVE-18891
 URL: https://issues.apache.org/jira/browse/HIVE-18891
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Amit Chauhan


Query : 

explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE 
COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM 
UDM_PRD_ITVM.SALES_PERSON) or SLS_PRSN_ID_NEW LIKE 'abc';
FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery 
Expression ''abc'': Only SubQuery expressions that are top level conjuncts are 
allowed.

 

 if i am running query in standalone like the 2 ways mentioned below they are 
running:

hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE 
COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM 
UDM_PRD_ITVM.SALES_PERSON) ;
OK

hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS a WHERE 
a.SLS_PRSN_ID_NEW LIKE 'abc' ;
OK

can you please help why the query with both conditions is not running. 

 

hiver version is :

hive> !hive --version;
Hive 1.2.1000.2.5.6.0-40

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18872) Projection is not pushed properly when query involves multiple tables

[Ankit Singhal (JIRA)]

Ankit Singhal created HIVE-18872:


 Summary: Projection is not pushed properly when query involves 
multiple tables
 Key: HIVE-18872
 URL: https://issues.apache.org/jira/browse/HIVE-18872
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ankit Singhal
Assignee: Ankit Singhal
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/
---

(Updated March 6, 2018, 5:24 p.m.)


Review request for hive, Peter Vary and Adam Szita.


Changes
---

Fix some review findings and also added some test cases for adding partitions 
to view.


Bugs: HIVE-18696
https://issues.apache.org/jira/browse/HIVE-18696


Repository: hive-git


Description
---

The idea behind the patch is

1) Separate the partition validation from starting the tasks which create the 
partition folders. 
Instead of doing the checks on the partitions and submit the tasks in one loop, 
separated the validation into a different loop. So first iterate through the 
partitions, validate the table/db names, and check for duplicates. Then if all 
partitions were correct, in the second loop submit the tasks to create the 
partition folders. This way if one of the partitions is incorrect, the 
exception will be thrown in the first loop, before the tasks are submitted. So 
we can be sure that no partition folder will be created if the list contains an 
invalid partition.

2) Handle the exceptions which occur during the execution of the tasks 
differently.
Previously if an exception occured in one task, the remaining tasks were 
canceled, and the newly created partition folders were cleaned up in the 
finally part. The problem was that it could happen that some tasks were still 
not finished with the folder creation when cleaning up the others, so there 
could have been leftover folders. After doing some testing it turned out that 
this use case cannot be avoided completely when canceling the tasks.
The idea of this patch is to set a flag if an exception is thrown in one of the 
tasks. This flag is visible in the tasks and if its value is true, the 
partition folders won't be created. Then iterate through the remaining tasks 
and wait for them to finish. The tasks which are started before the flag got 
set will then finish creating the partition folders. The tasks which are 
started after the flag got set, won't create the partition folders, to avoid 
unnecessary work. This way it is sure that all tasks are finished, when 
entering the finally part where the partition folders are cleaned up.


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 2be018b 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 4d9cb1b 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
 1122057 


Diff: https://reviews.apache.org/r/65716/diff/2/

Changes: https://reviews.apache.org/r/65716/diff/1-2/


Testing
---

Added some new tests cases to the TestAddPartitions and 
TestAddPartitionsFromPartSpec tests.


Thanks,

Marta Kuczora

Re: Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Marta Kuczora via Review Board]

> On Feb. 21, 2018, 3:32 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
> > Line 3032 (original), 3065 (patched)
> > 
> >
> > this code looks very similar to the block above. I know its was never 
> > the intention of this JIRA to do any re-factoring, but how difficult would 
> > it be to move all this code into a common method so that we don't have to 
> > fix the bug in two places? not a blocking issue though
> 
> Marta Kuczora wrote:
> Yeah, I absolutely agree. This code duplication annoys me as well, just I 
> wasn't sure that it is acceptable doing the refactoring in the scope of this 
> Jira. But it is not so difficult, so I will upload a patch where I moved the 
> common parts to a separate method and we can decide if it is ok like that or 
> rather do it in a different Jira.

I checked how this could be refactored and there are some differences between 
the methods which makes it not that straightforward. It is not that difficult 
and basically I have the patch, but I would do it in the scope of an other 
Jira, so we can discuss some details there. Would this be ok for you Sahil?


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/#review197829
---


On March 6, 2018, 5:24 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65716/
> ---
> 
> (Updated March 6, 2018, 5:24 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18696
> https://issues.apache.org/jira/browse/HIVE-18696
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The idea behind the patch is
> 
> 1) Separate the partition validation from starting the tasks which create the 
> partition folders. 
> Instead of doing the checks on the partitions and submit the tasks in one 
> loop, separated the validation into a different loop. So first iterate 
> through the partitions, validate the table/db names, and check for 
> duplicates. Then if all partitions were correct, in the second loop submit 
> the tasks to create the partition folders. This way if one of the partitions 
> is incorrect, the exception will be thrown in the first loop, before the 
> tasks are submitted. So we can be sure that no partition folder will be 
> created if the list contains an invalid partition.
> 
> 2) Handle the exceptions which occur during the execution of the tasks 
> differently.
> Previously if an exception occured in one task, the remaining tasks were 
> canceled, and the newly created partition folders were cleaned up in the 
> finally part. The problem was that it could happen that some tasks were still 
> not finished with the folder creation when cleaning up the others, so there 
> could have been leftover folders. After doing some testing it turned out that 
> this use case cannot be avoided completely when canceling the tasks.
> The idea of this patch is to set a flag if an exception is thrown in one of 
> the tasks. This flag is visible in the tasks and if its value is true, the 
> partition folders won't be created. Then iterate through the remaining tasks 
> and wait for them to finish. The tasks which are started before the flag got 
> set will then finish creating the partition folders. The tasks which are 
> started after the flag got set, won't create the partition folders, to avoid 
> unnecessary work. This way it is sure that all tasks are finished, when 
> entering the finally part where the partition folders are cleaned up.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  2be018b 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  4d9cb1b 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
>  1122057 
> 
> 
> Diff: https://reviews.apache.org/r/65716/diff/2/
> 
> 
> Testing
> ---
> 
> Added some new tests cases to the TestAddPartitions and 
> TestAddPartitionsFromPartSpec tests.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[jira] [Created] (HIVE-18873) Skipping predicate pushdown silently at HiveInputFormat can cause storage handlers to produce erroneous result

[Ankit Singhal (JIRA)]

Ankit Singhal created HIVE-18873:


 Summary: Skipping predicate pushdown silently at HiveInputFormat 
can cause storage handlers to produce erroneous result
 Key: HIVE-18873
 URL: https://issues.apache.org/jira/browse/HIVE-18873
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ankit Singhal
Assignee: Ankit Singhal
 Fix For: 3.0.0


{code:java}
// disable filter pushdown for mapreduce when there are more than one table 
aliases,

    // since we don't clone jobConf per alias

    if (mrwork != null && mrwork.getAliases() != null && 
mrwork.getAliases().size() > 1 &&

      jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {

      return;

    }
{code}
I believe this needs to be handled at OpProcFactory so that hive doesn't 
believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #316: HIVE-18864: ValidWriteIdList snapshot seems incorrec...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/316

HIVE-18864: ValidWriteIdList snapshot seems incorrect if obtained after 
allocating writeId by current transaction.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-18864

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/316.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #316


commit 85afddc2ab46d2f73f1ed7bcdb81105a30f7db8a
Author: Sankar Hariappan 
Date:   2018-03-06T17:20:50Z

HIVE-18864: ValidWriteIdList snapshot seems incorrect if obtained after 
allocating writeId by current transaction.




---

Re: Review Request 65756: HIVE-18060 UpdateInputAccessTimeHook fails for non-current database

[Oleksiy Sayankin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65756/
---

(Updated March 6, 2018, 7:42 p.m.)


Review request for hive, Ashutosh Chauhan and Zoltan Haindrich.


Changes
---

Added tests


Repository: hive-git


Description
---

initial commit


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 4a52eb5 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook.java 
c4856b1 
  ql/src/test/queries/clientpositive/update_access_time_non_current_db.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/update_access_time_non_current_db.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65756/diff/2/

Changes: https://reviews.apache.org/r/65756/diff/1-2/


Testing
---


Thanks,

Oleksiy Sayankin

Re: Review Request 65422: HIVE-17626

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated March 6, 2018, 11:06 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

11 ; address review comments; disable opstats collection for successfull first 
query ; added a conf variable to enable it for tests.


Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edea129579 
  data/conf/llap/hive-site.xml c4c299c5de 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 cbeac2c00a 
  itests/src/test/resources/testconfiguration.properties e8aa827523 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 07e6eaad88 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 6999777297 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
08b791ad42 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
131127e50d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 77e9263e0f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
1e28ca843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
2b345d6ec7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e422 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 8c1bcb3f62 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
 d1c53cf345 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 0c6aece1df 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
714cf3914b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
7332693513 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
5d3fdb8b63 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 
3837a49934 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2 
  

[GitHub] hive pull request #315: HIVE-18749: Need to replace transactionId with write...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/315


---

[jira] [Created] (HIVE-18870) Invalid GPG signature of the 2.3.2 release

[dud (JIRA)]

dud created HIVE-18870:
--

 Summary: Invalid GPG signature of the 2.3.2 release
 Key: HIVE-18870
 URL: https://issues.apache.org/jira/browse/HIVE-18870
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2
Reporter: dud


Hello

the [GPG 
signature|https://www.apache.org/dist/hive/hive-2.3.2/apache-hive-2.3.2-bin.tar.gz.asc]
 of the 2.3.2 Hive release is invalid preventing users to verify the downloaded 
archive :

 
{code:java}
gpg --primary-keyring /tmp/tmp.0xXkR5pkws.gpg --import ./hive-KEYS
gpg: key 085BCB4D2E765AE1: public key "Ashish Thusoo (CODE SIGNING KEY) 
" imported
gpg: key E36CD0AA4626191C: public key "Zheng Shao (Zheng Shao) 
" imported
gpg: key 0CA0FF40EDAC684E: public key "Carl W. Steinbach (CODE SIGNING KEY) 
" imported
gpg: key 2036CB247AC28520: public key "John Sichi (CODE SIGNING KEY) 
" imported
gpg: key E193B38C2305CA1C: public key "Ashutosh Chauhan " 
imported
gpg: key 5E1840C075E68997: public key "Thejas Nair (CODE SIGNING KEY) 
" imported
gpg: key 4814E367885D649D: public key "Harish Butani (CODE SIGNING KEY) 
" imported
gpg: key 1EADB7814530BADE: public key "Sushanth Sowmyan (CODE SIGNING KEY) 
" imported
gpg: key D90DF4E28EE6E329: public key "Thejas M Nair (CODE SIGNING KEY 2) 
" imported
gpg: key A645F24823724330: public key "Gunther Hagleitner (CODE SIGNING KEY) 
" imported
gpg: key 1209E7F13D0C92B9: 8 duplicate signatures removed
gpg: key 1209E7F13D0C92B9: 47 signatures not checked due to missing keys
gpg: key 1209E7F13D0C92B9: public key "Owen O'Malley (Code signing) 
" imported
gpg: key 88BD3F5704D9B832: public key "Alan Gates (No comment) 
" imported
gpg: key D0111CE83C2F98F8: public key "Sergio Pena " imported
gpg: key BBDD5B6642FE69CE: public key "stakiar " 
imported
gpg: Total number processed: 14
gpg:   imported: 14
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   4  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   4  signed:   0  trust: 2-, 0q, 0n, 2m, 0f, 0u
gpg: next trustdb check due at 2022-07-10
gpg --primary-keyring /tmp/tmp.0xXkR5pkws.gpg --verify 
/tmp/hive/apache-hive-2.3.2-bin.tar.gz.asc 
/tmp/hive/apache-hive-2.3.2-bin.tar.gz
gpg: Signature made jeu. 09 nov. 2017 20:32:14 CET
gpg:    using RSA key B21AE8EEA5105E6CA3F9EFA1BBDD5B6642FE69CE
gpg: BAD signature from "stakiar " [unknown]
{code}
I've already contacted [~stakiar] some weeks ago about this issue but 
unfortunatelely the signature hasn't been fixed yet.

Regards

dud



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://

[zhuwei (JIRA)]

zhuwei created HIVE-18871:
-

 Summary: hive on tez execution error due to set hive.aux.jars.path 
to hdfs://
 Key: HIVE-18871
 URL: https://issues.apache.org/jira/browse/HIVE-18871
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: 2.2.1
Reporter: zhuwei
Assignee: zhuwei


When set the properties 
hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar

and hive.execution.engine=tez; execute any query will fail with below error log:

exec.Task: Failed to execute tez graph.
java.lang.IllegalArgumentException: Wrong FS: 
hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:///
 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) 
~[hadoop-common-2.6.0.jar:?]
 at 
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) 
~[hadoop-common-2.6.0.jar:?]
 at 
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529)
 ~[hadoop-common-2.6.0.jar:?]
 at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
 ~[hadoop-common-2.6.0.jar:?]
 at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
 ~[hadoop-common-2.6.0.jar:?]
 at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) 
~[hadoop-common-2.6.0.jar:?]
 at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) 
~[hadoop-common-2.6.0.jar:?]
 at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) 
~[hadoop-common-2.6.0.jar:?]
 at 
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007)
 ~[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) 
~[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845)
 ~[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466)
 ~[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252)
 ~[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622)
 ~[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
 ~[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) 
~[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) 
[hive-exec-2.1.1.jar:2.1.1]
 at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) 
[hive-exec-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) 
[hive-cli-2.1.1.jar:2.1.1]
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) 
[hive-cli-2.1.1.jar:2.1.1]
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_111]
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_111]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
 at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
[hadoop-common-2.6.0.jar:?]
 at 

Re: Review Request 65731: HIVE-18699: Check for duplicate partitions in HiveMetastore.exchange_partitions

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65731/#review198704
---


Ship it!




Ship It!

- Peter Vary


On Feb. 21, 2018, 11:37 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65731/
> ---
> 
> (Updated Feb. 21, 2018, 11:37 a.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18699
> https://issues.apache.org/jira/browse/HIVE-18699
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the HiveMetastore.exchange_partitions method to check if the 
> partitions to be exchanged don't exist in the dest table. If one of the 
> partitions already exists, throw a MetaException with a proper error message.
> 
> Previously an exception like this (wrapped in a MetaException) was thrown:
> Insert of object
> "org.apache.hadoop.hive.metastore.model.MPartition@4e78fff5" using statement 
> "INSERT INTO PARTITIONS
> (PART_ID,CREATE_TIME,LAST_ACCESS_TIME,PART_NAME,SD_ID,TBL_ID) VALUES 
> (?,?,?,?,?,?)" failed : The statement was
> aborted because it would have caused a duplicate key value in a unique or 
> primary key constraint or unique index
> identified by 'UNIQUEPARTITION' defined on 'PARTITIONS'.
> 
> From user point of view, the type of the exception is not changed 
> (MetaException), just the error message is changed to a more understandable 
> one.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  3a06aec 
> 
> 
> Diff: https://reviews.apache.org/r/65731/diff/1/
> 
> 
> Testing
> ---
> 
> Tests already exist for this use case in TestExchangePartitions:
> - testExchangePartitionsPartAlreadyExists
> - testExchangePartitionPartAlreadyExists
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65731: HIVE-18699: Check for duplicate partitions in HiveMetastore.exchange_partitions

[Marta Kuczora via Review Board]

> On Feb. 21, 2018, 1:37 p.m., Peter Vary wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
> > Lines 3370 (patched)
> > 
> >
> > How "expensive" is this call? Is this a simple query? What happens if 
> > the destintaion table has 1m partitions? :)
> 
> Adam Szita wrote:
> I don't see any other way (currently available) that is more lightweight 
> than this is. Under the hood this calls getPartitionNamesNoTxn which executes 
> an actual "select parititionName from .." statement.

Yeah, as Adam says it is the least expensive way to do this check. It would 
mean one extra "select partitionName" per exchangePartition calls.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65731/#review197848
---


On Feb. 21, 2018, 11:37 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65731/
> ---
> 
> (Updated Feb. 21, 2018, 11:37 a.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18699
> https://issues.apache.org/jira/browse/HIVE-18699
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the HiveMetastore.exchange_partitions method to check if the 
> partitions to be exchanged don't exist in the dest table. If one of the 
> partitions already exists, throw a MetaException with a proper error message.
> 
> Previously an exception like this (wrapped in a MetaException) was thrown:
> Insert of object
> "org.apache.hadoop.hive.metastore.model.MPartition@4e78fff5" using statement 
> "INSERT INTO PARTITIONS
> (PART_ID,CREATE_TIME,LAST_ACCESS_TIME,PART_NAME,SD_ID,TBL_ID) VALUES 
> (?,?,?,?,?,?)" failed : The statement was
> aborted because it would have caused a duplicate key value in a unique or 
> primary key constraint or unique index
> identified by 'UNIQUEPARTITION' defined on 'PARTITIONS'.
> 
> From user point of view, the type of the exception is not changed 
> (MetaException), just the error message is changed to a more understandable 
> one.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  3a06aec 
> 
> 
> Diff: https://reviews.apache.org/r/65731/diff/1/
> 
> 
> Testing
> ---
> 
> Tests already exist for this use case in TestExchangePartitions:
> - testExchangePartitionsPartAlreadyExists
> - testExchangePartitionPartAlreadyExists
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65730: HIVE-18697: The HiveMetastore.exchange_partitions method throws FileNotFoundException if the given partition doesn't exist in the source table

[Marta Kuczora via Review Board]

> On Feb. 21, 2018, 1:32 p.m., Peter Vary wrote:
> > Ship It!

Thanks a lot Peter for the review!


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65730/#review197847
---


On Feb. 21, 2018, 10:36 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65730/
> ---
> 
> (Updated Feb. 21, 2018, 10:36 a.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18697
> https://issues.apache.org/jira/browse/HIVE-18697
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the HiveMetastore.exchange_partitions method to check if the 
> partitionsToExchange list is empty and if it is throw a MetaException with a 
> proper error message that no partition exists with the given values for the 
> source table. Previously a FileNotFoundException was thrown (wrapped in a 
> MetaException) when tried to move the partition folder to the dest table. So 
> the type of the exception was not changed, only the error message.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  3a06aec 
> 
> 
> Diff: https://reviews.apache.org/r/65730/diff/1/
> 
> 
> Testing
> ---
> 
> There are tests for this use case in TestExchangePartitions:
> - testExchangePartitionsNoPartExists
> - testExchangePartitionNoPartExists
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65731: HIVE-18699: Check for duplicate partitions in HiveMetastore.exchange_partitions

[Marta Kuczora via Review Board]

> On Feb. 22, 2018, 10:35 a.m., Adam Szita wrote:
> > Ship It!

Thanks a lot Adam for the review!


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65731/#review198106
---


On Feb. 21, 2018, 11:37 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65731/
> ---
> 
> (Updated Feb. 21, 2018, 11:37 a.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18699
> https://issues.apache.org/jira/browse/HIVE-18699
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the HiveMetastore.exchange_partitions method to check if the 
> partitions to be exchanged don't exist in the dest table. If one of the 
> partitions already exists, throw a MetaException with a proper error message.
> 
> Previously an exception like this (wrapped in a MetaException) was thrown:
> Insert of object
> "org.apache.hadoop.hive.metastore.model.MPartition@4e78fff5" using statement 
> "INSERT INTO PARTITIONS
> (PART_ID,CREATE_TIME,LAST_ACCESS_TIME,PART_NAME,SD_ID,TBL_ID) VALUES 
> (?,?,?,?,?,?)" failed : The statement was
> aborted because it would have caused a duplicate key value in a unique or 
> primary key constraint or unique index
> identified by 'UNIQUEPARTITION' defined on 'PARTITIONS'.
> 
> From user point of view, the type of the exception is not changed 
> (MetaException), just the error message is changed to a more understandable 
> one.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  3a06aec 
> 
> 
> Diff: https://reviews.apache.org/r/65731/diff/1/
> 
> 
> Testing
> ---
> 
> Tests already exist for this use case in TestExchangePartitions:
> - testExchangePartitionsPartAlreadyExists
> - testExchangePartitionPartAlreadyExists
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: HMS API testing and direct SQL

[Peter Vary]

Hi Team,

The MetaStore API tests are designed to be easily extensible by new MetaStore 
configurations. Just take a look at the following class: 
MetaStoreFactoryForTests. We can add a new configurations here like it is done 
with the RemoteMetaStore.
What we have to decide is: which is the right level to test DirectSQL methods. 
MetaStore level or on ObjectStore level. I can see pros and cons about both 
solutions. What do you think?

Thanks,
Peter

> On Mar 2, 2018, at 8:14 PM, Sergey Shelukhin  wrote:
> 
> There used to be a special ObjectStore for tests that would run stuff both
> with and without DirectSql in tests and fail on mismatch. Not sure about
> its state now.
> 
> On 18/3/2, 11:05, "Vihang Karajgaonkar"  wrote:
> 
>> Direct SQL is controlled by configs metastore.try.direct.sql
>> and metastore.try.direct.sql.ddl which are enabled by default. So I am
>> guessing the API tests will cover the direct SQL enabled path only unless
>> the tests override the default configs (I don't think they do).
>> 
>> On Fri, Mar 2, 2018 at 10:51 AM, Alexander Kolbasov 
>> wrote:
>> 
>>> We currently use two distinct code paths via ObjectStore - the regular
>>> one
>>> and direct SQL. So I think all the tests that verify HMS API
>>> functionality
>>> should somehow cover both cases.
>>> 
>>> Peter - what are your thoughts?
>>> 
>>> - Alex
>>> 
>