[jira] [Created] (HIVE-21260) Hive 3 (onprem) -> 4(onprem): Hive replication failed due to postgres sql execution issue

[mahesh kumar behera (JIRA)]

mahesh kumar behera created HIVE-21260:
--

 Summary: Hive 3 (onprem) -> 4(onprem): Hive replication failed due 
to postgres sql execution issue
 Key: HIVE-21260
 URL: https://issues.apache.org/jira/browse/HIVE-21260
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera
 Fix For: 4.0.0


Missing quotes in sql string is causing sql execution error for postgres.

 
{code:java}
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - 
MetaException(message:Unable to update transaction database 
org.postgresql.util.PSQLException: ERROR: relat
ion "database_params" does not exist
Position: 25
at 
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
at 
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:284)
at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
at 
com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.updateReplId(TxnHandler.java:907)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(TxnHandler.java:1023)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.commit_txn(HiveMetaStore.java:7703)
at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy39.commit_txn(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:18730)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:18714)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
){code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21259) HiveMetaStoreCilent.getNextNotification throws exception when no new events found

[anishek (JIRA)]

anishek created HIVE-21259:
--

 Summary: HiveMetaStoreCilent.getNextNotification throws exception 
when no new events found
 Key: HIVE-21259
 URL: https://issues.apache.org/jira/browse/HIVE-21259
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: anishek
Assignee: anishek
 Fix For: 4.0.0


HiveMetastoreClient can be used to get next notifications for hiveserver2. if 
the user has received all notifications and there were no new notifications 
generated on the server, then the user should get an empty response, however 
currently the client throws an IllegalStateException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21258) Streamline and Add Warning to PrimitiveObjectInspectorFactory.java

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-21258:
--

 Summary: Streamline and Add Warning to 
PrimitiveObjectInspectorFactory.java
 Key: HIVE-21258
 URL: https://issues.apache.org/jira/browse/HIVE-21258
 Project: Hive
  Issue Type: Improvement
Affects Versions: 4.0.0, 3.2.0
Reporter: BELUGA BEHR


I just got bit by something pretty good, so I would like to propose adding a 
developer-facing warning into the logs to avoid this situation again.  Also, 
fix up the related cache a bit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21256) Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in Hive 2

[Teddy Choi (JIRA)]

Teddy Choi created HIVE-21256:
-

 Summary: Vectorized LENGTH UDF doesn't respect the max length of 
VARCHAR or CHAR in Hive 2
 Key: HIVE-21256
 URL: https://issues.apache.org/jira/browse/HIVE-21256
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.4
Reporter: Teddy Choi
Assignee: Teddy Choi


After HIVE-19951 is fixed, there still are some cases that vectorized length 
UDF doesn't respect the max length of VARCHAR or CHAR. StringLength has an 
internal bug. Moreover, it's hard to get input data type details in Hive 2, 
unlike Hive 3. So separate both implementation to keep code clean in Hive 3 
while the changes minimal in Hive 2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21257) Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in Hive 3+

[Teddy Choi (JIRA)]

Teddy Choi created HIVE-21257:
-

 Summary: Vectorized LENGTH UDF doesn't respect the max length of 
VARCHAR or CHAR in Hive 3+
 Key: HIVE-21257
 URL: https://issues.apache.org/jira/browse/HIVE-21257
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.1, 4.0.0
Reporter: Teddy Choi
Assignee: Teddy Choi


After HIVE-19951 is fixed, there still are some cases that vectorized length 
UDF doesn't respect the max length of VARCHAR or CHAR. StringLength has an 
internal bug. Moreover, it's hard to get input data type details in Hive 2, 
unlike Hive 3. So separate both implementation to keep code clean in Hive 3 
while the changes minimal in Hive 2.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21255:
-

 Summary: Remove QueryConditionBuilder in JdbcStorageHandler
 Key: HIVE-21255
 URL: https://issues.apache.org/jira/browse/HIVE-21255
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai


QueryConditionBuilder is not correctly implemented. Always see the following 
exception even the query finish succeeded:
{code}
2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|: Error 
during condition build
java.lang.ArrayIndexOutOfBoundsException: 0
at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
at 
org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
at 
org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
at 
org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
at 
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
We don't actually need QueryConditionBuilder when cbo is enabled since 
predicate push down is handled by calcite (HIVE-20822). One can argue when cbo 
is disabled we might still need that since calcite will not do the push down, 
but that's a minor code path and removing QueryConditionBuilder won't cause any 
correctness issue. So I'd like to remove it for simplicity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69965: HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69965/
---

(Updated Feb. 13, 2019, 1:15 a.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Changes
---

Addressed review comments.


Bugs: HIVE-21254
https://issues.apache.org/jira/browse/HIVE-21254


Repository: hive-git


Description
---

HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables


Diffs (updated)
-

  
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java
 04782a62ad 
  
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java
 fe4b08b3f3 


Diff: https://reviews.apache.org/r/69965/diff/2/

Changes: https://reviews.apache.org/r/69965/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 69965: HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69965/
---

Review request for hive, Eugene Koifman and Jason Dere.


Bugs: HIVE-21254
https://issues.apache.org/jira/browse/HIVE-21254


Repository: hive-git


Description
---

HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables


Diffs
-

  
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java
 04782a62ad9babb6f2b24ab76c6210a27bdfdceb 
  
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java
 fe4b08b3f360faba52172fecb97e246cc7acb4a1 


Diff: https://reviews.apache.org/r/69965/diff/1/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-21254:


 Summary: Pre-upgrade tool should handle exceptions and skip 
db/tables
 Key: HIVE-21254
 URL: https://issues.apache.org/jira/browse/HIVE-21254
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 4.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


When exceptions like AccessControlException is thrown, pre-upgrade tool fails. 
If hive user does not have read access to database or tables (some external 
tables denies read access to hive), pre-upgrade tool should just assume they 
are external tables and move on without failing pre-upgrade process. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

RE: Setting up a Standalone Hive Metastore

[Joo Wan Ro]

Please disregard.. I think I have it successfully running when I switched to 
Ubuntu.  I ran into a lot of funky issues when running it on Windows, as shown 
in my previous email.

Joo Wan

From: Joo Wan Ro 
Sent: Monday, February 11, 2019 11:31 AM
To: u...@hive.apache.org; dev@hive.apache.org
Cc: Karthick Krishnamoorthy ; Danny Navon 

Subject: RE: Setting up a Standalone Hive Metastore

[Resending because I needed to add myself to the mailing lists]

Hello Hive Experts,

I am a software engineer at Microsoft, and I am having trouble trying to run a 
standalone Hive metastore service on my Windows 10 machine.  Your assistance 
would be greatly appreciated.

[0] Github project here (running branch "3.1"): 
link
[1] Related documentation here: 
link
[2] Path to setup scripts: ($HOME\hive\standalone-metastore\src\main\scripts)

I was able to build it successfully using Maven but having trouble running the 
jar.  Here are my questions:


  1.  I believe this standalone metastore project is relatively new, as it was 
introduced in Hive 3.0.  When was the stable version of the standalone 
metastore released?



  1.  While the documentation link [1] above helps with the config setup, is 
there a resource for step-by-step guidance on how to bootstrap the standalone 
metastore service on a Windows 10 or Linux machine?  Without it, I've been 
trying to reverse engineer on how to get it running.



 *   For starters, it seems that the scripts under the directory above [2] 
are not Windows 10 friendly because of carriage returns.  And having to using 
Cygwin confused me on which path convention to use for the environment/system 
variables (E:\src\hive\... vs. /cygdrive/e/src/hive/).  I removed the carriage 
returns and used the /cygdrive/ convention to get it working partially.



 *   I had no clue which environment/system variables I needed and if there 
were any dependencies, which I assumed to be none because the related 
documentation [1] above notes the independent nature of the standalone 
metastore project.  However, by studying the scripts (base, start-metastore, 
and metastore.sh) under the path above [2], I found two things:



   i.  The need 
to define METASTORE_HOME 
($HOME\hive\standalone-metastore\target\apache-hive-metastore-3.1.1-bin\apache-hive-metastore-3.1.1-bin)
 and METASTORE_CONF_DIR environment variables



 ii.  The need 
to install Hadoop as a dependency because the metastore.sh script uses it to 
start the metastore service; hence, installing it and then defining the 
HADOOP_HOME ($HOME\hadoop-3.1.1) environment variable (I also had to remove the 
carriage returns under $HOME\hadoop-3.1.1)



   iii.  I have no 
other environment variables or dependencies other than the ones aforementioned



 *   At this point, the metastore service began to start running; however, 
I ran into an exception "Failed to get schema version" - more information here: 
link.
  I believe this is because the default derby database was not initialized.



 *   So, using the schematool script under my 
apache-hive-metastore-3.1.1-bin directory, I ran schematool --dbType derby 
-initSchema.  Then I ran into an exception "Unknown version specified for 
initialization: 3.1.0" - the exception is thrown here 
link.
  It cannot find the derby schema script, but I 

[jira] [Created] (HIVE-21253) Support DB2 in JDBC StorageHandler

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21253:
-

 Summary: Support DB2 in JDBC StorageHandler
 Key: HIVE-21253
 URL: https://issues.apache.org/jira/browse/HIVE-21253
 Project: Hive
  Issue Type: Improvement
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


Make DB2 a first class member of JdbcStorageHandler. It can even work before 
the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This 
patch make it a standard feature. Note this is only for DB2 tables as external 
JdbcStorageHandler table. We haven't tested DB2 as a metastore backend and it's 
not a goal for this ticket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69918: HIVE-21001 Update to Calcite 1.18

[Ashutosh Chauhan]

> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > accumulo-handler/src/test/results/positive/accumulo_predicate_pushdown.q.out
> > Line 417 (original), 417 (patched)
> > 
> >
> > This should further fold to key >= '90'Can you file a follow-up jira 
> > for this?
> 
> Zoltan Haindrich wrote:
> I think this might not be simplified further; because when key is null 
> the expression should be true.
> ```
> (key < '90') is not true
> (key >= '90') or key is null
> 
> not COALESCE((key < '90'),false)
> COALESCE((key >= '90'),true)
> ```

Thats right. But it can be folded to key < '90' is false


> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constant_prop_3.q.out
> > Line 286 (original), 286 (patched)
> > 
> >
> > New expression tree is longer compared to original. I guess we try to 
> > apply DeMorgan theorem here, but in this case its a net loss. Perhaps, we 
> > can add a (simple) logic which says if node count in expression tree grows 
> > after the application of theorem we throw away that.
> 
> Zoltan Haindrich wrote:
> simplification is too conservative in 1.18; see: CALCITE-2840

We shall make CALCITE-2840 blocker for 1.19 release since its a regression.


> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/fold_case.q.out
> > Line 21 (original), 21 (patched)
> > 
> >
> > {{is true}} is redundant. Can we re-fold it?
> 
> Zoltan Haindrich wrote:
> yes; simplify currently not removes redundant IS X w.r.t. unknownAs mode 
> - I've noted this somewhere...
> 
> it's interesting that it earlier worked
> CALCITE-2838

We shall make CALCITE-2838 blocker for 1.19 release since its a regression.


> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out
> > Line 50 (original), 50 (patched)
> > 
> >
> > New Expression is more complex to evaluate. Can we refold this?
> 
> Zoltan Haindrich wrote:
> yes, this shouldn't happen; "ELSE NULL" should really be "ELSE FALSE"

is this tracked in a jira?


> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/fold_when.q.out
> > Line 227 (original), 227 (patched)
> > 
> >
> > This {{is true}} also gets in a way of constant propagation.
> 
> Zoltan Haindrich wrote:
> CALCITE-2838

We shall make CALCITE-2838 blocker for 1.19 release since its a regression.


> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/tez/query70.q.out
> > Line 113 (original), 113 (patched)
> > 
> >
> > UDFToLong(0) should be folded. Can you file a follow-up jira for it?
> 
> Zoltan Haindrich wrote:
> yes; cast(null as string) also seems to be odd
> at the ast level it looks good - calcite doesn't seem to be leaving an 
> explicit cast there

is this tracked in a jira?


> On Feb. 7, 2019, 9:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/ppd_udf_case.q.out
> > Line 50 (original), 49 (patched)
> > 
> >
> > New expression is more expensive to evaluate.
> 
> Zoltan Haindrich wrote:
> this expression is false
> ```
> (null and (key = '27') is not true and (key = '38') is not true)) is true
> ```
> I'll take a look why it regressed
> 
> after some calcite fixes I've prepared so far...most probably CALCITE-2840

We shall make CALCITE-2840 blocker for 1.19 release since its a regression.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69918/#review212637
---


On Feb. 7, 2019, 7:08 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69918/
> ---
> 
> (Updated Feb. 7, 2019, 7:08 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-21001
> https://issues.apache.org/jira/browse/HIVE-21001
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> patch#1 here is #23 on jira
> 
> 
> Diffs
> -
> 
>   
> accumulo-handler/src/test/results/positive/accumulo_predicate_pushdown.q.out 
> 

[jira] [Created] (HIVE-21252) LazyTimestamp - Use String Equals

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-21252:
--

 Summary: LazyTimestamp - Use String Equals
 Key: HIVE-21252
 URL: https://issues.apache.org/jira/browse/HIVE-21252
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 4.0.0, 3.2.0
Reporter: BELUGA BEHR


{code:java|title=LazyTimestamp.java}
if (s.compareTo("NULL") == 0) {
  isNull = true;
  logExceptionMessage(bytes, start, length, "TIMESTAMP");
}
{code}

compareTo generates a number to represent the differences between the two 
Strings.  It's faster to simply call "equals" which will simply compare the two 
String directly and return a boolean.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69918: HIVE-21001 Update to Calcite 1.18

[Zoltan Haindrich]

> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > accumulo-handler/src/test/results/positive/accumulo_predicate_pushdown.q.out
> > Line 417 (original), 417 (patched)
> > 
> >
> > This should further fold to key >= '90'Can you file a follow-up jira 
> > for this?

I think this might not be simplified further; because when key is null the 
expression should be true.
```
(key < '90') is not true
(key >= '90') or key is null

not COALESCE((key < '90'),false)
COALESCE((key >= '90'),true)
```


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java
> > Lines 156 (patched)
> > 
> >
> > Can you add a comment for this, since this is counter-intuitive.

sure. This enables an agressive project merge - which is done by the builder 
and it merges together multiple windowing expressions ; there are rules to 
merge projects together - this builder feature looks like something which is 
somewhat redundant


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/cbo_rp_simple_select.q.out
> > Line 902 (original), 900 (patched)
> > 
> >
> > We lost filterExpr on TableScan this will prohibit pushing of filters 
> > to ORC. Can you investigate this and file follow-up jira for this?

yes the filter is false; opened HIVE-21251


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constant_prop_3.q.out
> > Line 286 (original), 286 (patched)
> > 
> >
> > New expression tree is longer compared to original. I guess we try to 
> > apply DeMorgan theorem here, but in this case its a net loss. Perhaps, we 
> > can add a (simple) logic which says if node count in expression tree grows 
> > after the application of theorem we throw away that.

simplification is too conservative in 1.18; see: CALCITE-2840


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constprog_when_case.q.out
> > Line 53 (original), 53 (patched)
> > 
> >
> > Same comment as previous.

this is not simplified right now; because it's inside an "if" - I believe the 
best would be to rewrite IF to CASE before running calcite (HIVE-21152)

there was a problematic simplification invcation which visited every 
subexpression ; and it was prone to some bugs - the fix is to be more carefull; 
and as such it not visits "unknown udfs"


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/fold_case.q.out
> > Line 21 (original), 21 (patched)
> > 
> >
> > {{is true}} is redundant. Can we re-fold it?

yes; simplify currently not removes redundant IS X w.r.t. unknownAs mode - I've 
noted this somewhere...

it's interesting that it earlier worked
CALCITE-2838


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out
> > Line 50 (original), 50 (patched)
> > 
> >
> > New Expression is more complex to evaluate. Can we refold this?

yes, this shouldn't happen; "ELSE NULL" should really be "ELSE FALSE"


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/fold_when.q.out
> > Line 227 (original), 227 (patched)
> > 
> >
> > This {{is true}} also gets in a way of constant propagation.

CALCITE-2838


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/list_bucket_dml_14.q.out
> > Line 300 (original), 300 (patched)
> > 
> >
> > Although this doesn't affect correctness, but Hive does make a 
> > difference between string and varchar. It would have been useful to retain 
> > this cast as string since that is what is executed by Hive.

this seem have worked very similarily earlier; only the printout is "fixed"

https://github.com/apache/hive/blob/e540cf8238680453fc04e861dbc332e7f56b904a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1494

CALCITE-2713 added a check to limit it to 64K

I've set hive typesystem...it was the default during optimizedsql generation


> On Feb. 7, 2019, 10:16 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query78.q.out
> > Lines 150-154 (original), 147-160 (patched)
> > 

[jira] [Created] (HIVE-21251) Handle "false" filter predicate on tablescan

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-21251:
---

 Summary: Handle "false" filter predicate on tablescan
 Key: HIVE-21251
 URL: https://issues.apache.org/jira/browse/HIVE-21251
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


If the filter turned out to be false; the ts filter predicate is not "false"
So the reader may still read the data for the filter which will just throw away 
all rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] cricket007 commented on a change in pull request #526: HIVE-21218: KafkaSerDe doesn't support topics created via Confluent

[GitBox]

cricket007 commented on a change in pull request #526: HIVE-21218: KafkaSerDe 
doesn't support topics created via Confluent
URL: https://github.com/apache/hive/pull/526#discussion_r255860546
 
 

 ##
 File path: kafka-handler/pom.xml
 ##
 @@ -114,8 +114,21 @@
   1.7.25
   test
 
+
+  io.confluent
+  kafka-streams-avro-serde
 
 Review comment:
   That's just a serializer, not a Kafka **Streams** serde.
   
   You can relax the dependency to `kafka-avro-serializer` instead here. That's 
what I'm saying 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] xbaran commented on a change in pull request #526: HIVE-21218: KafkaSerDe doesn't support topics created via Confluent

[GitBox]

xbaran commented on a change in pull request #526: HIVE-21218: KafkaSerDe 
doesn't support topics created via Confluent
URL: https://github.com/apache/hive/pull/526#discussion_r255843661
 
 

 ##
 File path: kafka-handler/pom.xml
 ##
 @@ -114,8 +114,21 @@
   1.7.25
   test
 
+
+  io.confluent
+  kafka-streams-avro-serde
 
 Review comment:
   Yes, I'm using it in `AvroBytesConverterTest` class and it is 
`KafkaAvroSerializer avroSerializer = new KafkaAvroSerializer(new 
MockSchemaRegistryClient());`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rmsmani opened a new pull request #532: HIVE-685 GenericUDFQuote

[GitBox]

rmsmani opened a new pull request #532: HIVE-685 GenericUDFQuote
URL: https://github.com/apache/hive/pull/532
 
 
   HIVE-685 GenericUDFQuote
   @pvary Kindly review the source and merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services