Re: 3.1.4 release?

[Matt Burgess]

Naveen,

I don't think I have permissions to commit to branches? Way back when
I submitted the patch I think it was Gerrit or something, do I have to
do the same thing but for the branch-3?

Thank you,
Matt

On Thu, May 19, 2022 at 2:30 PM Naveen Gangam
 wrote:
>
> Hi Matt,
> There are some other pending CVEs to be addressed (like thrift upgrade) by
> upgrading the binaries we build on. So the plan is to do more releases.
> Please commit the change to the branch-3 as you see fit.
>
> Thank you
> Naveen
>
> On Fri, May 6, 2022 at 11:33 AM Matt Burgess  wrote:
>
> > Hi all,
> >
> > Are there plans for a 3.1.4 release? I noticed HIVE-22393 [1] didn't
> > get into Hive 3.1.3, but that bug makes Hive Streaming unusable, at
> > least in Apache NiFi. Is there a workaround, some property or
> > something I can set to make the stats work (or at least not throw an
> > NPE)?
> >
> > Thanks,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/HIVE-22393
> >

3.1.4 release?

[Matt Burgess]

Hi all,

Are there plans for a 3.1.4 release? I noticed HIVE-22393 [1] didn't
get into Hive 3.1.3, but that bug makes Hive Streaming unusable, at
least in Apache NiFi. Is there a workaround, some property or
something I can set to make the stats work (or at least not throw an
NPE)?

Thanks,
Matt

[1] https://issues.apache.org/jira/browse/HIVE-22393

[jira] [Created] (HIVE-24077) Streaming Ingest v2 does not send heartbeats

[Matt Burgess (Jira)]

Matt Burgess created HIVE-24077:
---

 Summary: Streaming Ingest v2 does not send heartbeats
 Key: HIVE-24077
 URL: https://issues.apache.org/jira/browse/HIVE-24077
 Project: Hive
  Issue Type: Bug
  Components: Streaming
Reporter: Matt Burgess


In Hive Streaming v2, TransactionBatch calls setupHeartbeatThread() before 
setting minTxnId to an AtomicLong object, so the value that is used for 
HeartbeatRunnable is null. Upon the first call to run(), a NullPointerException 
is thrown by the minTxnId.get() call, which causes the thread to fail and it 
does not get rescheduled. This causes any transaction that exceeds the 
transaction timeout time (default 5 mins) to be aborted.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

3.1.3 release?

[Matt Burgess]

Hello all,

Will there be a 3.1.3 release? If so can we get
https://issues.apache.org/jira/browse/HIVE-20979 in there? The Fix
Version says 4.0.0 and 3.1.1 but it is not in 3.1.1 or 3.1.2 (but is
in master). The memory leak can be fairly substantial for example in
the Apache NiFi Hive 3 processor PutHive3Streaming.

Thank you in advance,
Matt

Hive JDBC complex type objects

[Matt Burgess]

According to https://issues.apache.org/jira/browse/HIVE-1378 Hive
returns strings for complex types. I couldn’t find anything else on
the subject, has there been any more work done in later versions of
Hive to return actual objects in the JDBC driver? In NiFi we can have
records/results/rows with complex types, but when we translate a Hive
ResultSet into fields, complex types like arrays simply become Strings
when IMO they’d be more useful as actual arrays, for downstream
transformations etc.

Thank you in advance,
Matt

[jira] [Created] (HIVE-19741) Update documentation to reflect list of reserved words

[Matt Burgess (JIRA)]

Matt Burgess created HIVE-19741:
---

 Summary: Update documentation to reflect list of reserved words
 Key: HIVE-19741
 URL: https://issues.apache.org/jira/browse/HIVE-19741
 Project: Hive
  Issue Type: Improvement
  Components: Documentation
Reporter: Matt Burgess


The current list of non-reserved and reserved keywords is on the Hive wiki:

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords

However it does not match the list in code (see the lexer rules here):

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g

On particular example is the "application" keyword, which was discovered while 
trying to create a table with a column named "application".

This Jira proposes to align the documentation with the current set of 
non-reserved and reserved keywords.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19534) Allow implementations to access member variables of AbstractRecordWriter

[Matt Burgess (JIRA)]

Matt Burgess created HIVE-19534:
---

 Summary: Allow implementations to access member variables of 
AbstractRecordWriter
 Key: HIVE-19534
 URL: https://issues.apache.org/jira/browse/HIVE-19534
 Project: Hive
  Issue Type: Improvement
  Components: Streaming
Reporter: Matt Burgess


The AbstractRecordWriter class in the Hive 3 Streaming API (package 
org.apache.hive.streaming) provides common functionality for processing 
incoming records (each as a byte[]) where subclasses often need only to 
implement the encode() and createSerde() methods and let AbstractRecordWriter 
handle the rest.

However for some custom RecordWriters, the records may not be available as a 
byte array, and thus the custom RecordWriter may need to handle the writes and 
the "paperwork" such as connection stats updates (number of records written, 
e.g.), basically the same code that is in the write(long, byte[]) method. To do 
that, the subclass will need access to the member variables of 
AbstractRecordWriter, which are currently private. The same likely holds for 
the private methods.

This Jira proposes to make the member variables and methods of 
AbstractRecordWriter protected (or package-protected) as prudent. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Integration testing for clients

[Matt Burgess]

I'm looking into writing integration tests for our Hive processors in
Apache NiFi to be able to test against a "real" Hive Server 2, Hive
Metastore, etc.  I see in Hive 3.0 there's a MiniHMS and other test
classes, but I couldn't find them in a Maven repo anywhere.  Are there
artifacts published somewhere that I could use for integration
testing? If not, it might be a good idea to make those available for
Hive clients to test with, especially if it doesn't require a full
Hive install.

Thanks,
Matt

Re: Hive unit testing in other projects

[Matt Burgess]

I am interested in this as well, we have Hive processors (in Apache
NiFi) but have been using Derby to test the processor logic (not
necessarily the interaction with Hive). With an embedded/small Hive it
would be an integration test but still would help a great deal to iron
out bugs.

Regards,
Matt

On Tue, Aug 9, 2016 at 7:37 PM, Chris Teoh  wrote:
> Hi folks,
>
> I'm working on a Sqoop patch that imports to Hive and am wondering what's
> the easiest way to incorporate a unit test that uses a mini Hive server
> without requiring a full dependency on Hive project?
>
> Kind Regards
> Chris

Unions of Maps

[Matt Burgess]

Hello all,

I'm using Hive 1.2.1, and let's say I have the following type info
(code simplified):

typeInfo = 
TypeInfoUtils.getTypeInfoFromTypeString("uniontype,map>")

I can get the appropriate ObjectInspectors:

oi = TypeInfoUtils.getStandardWritableObjectInspectorFromTypeInfo(typeInfo)

map1oi = oi.getObjectInspectors().get(0)

map2oi = oi.getObjectInspectors().get(1)

I'm trying to fill a map (actually a MapWritable, I'm converting these
to ORC).  Now let's say I'm given a Map object. How would I
programmatically tell which one of the union "entries" is the right
one to use? Do I just try the put() command and if it fails, try the
next ObjectInspector? Do I try to get the Class of the key/value and
recursively try to build an "output object" that way? Or am I missing
a better way to resolve the correct union type against the incoming
object?

Thank you in advance,
Matt

Double objects in ResultSet with column type Float

[Matt Burgess]

Hello all,

I had a question about the Hive JDBC driver and its treatment of
float/double types.  According to the following, floating-point
literals are treated as doubles:

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-FloatingPointTypes.

So let's say I have a column in position 2 of "float" type and I do an
insert that sets the column's value for that row to 1.0.  If I do a
executeQuery from the JDBC driver and call ResultSet.getObject(2), I
get a Double back even though if I do a
ResultSet.getMetaData().getColumnType(2) I get a 6
(java.sql.Types.FLOAT). If I call getFloat(2), I get a Float object
back.

Is this the intended behavior? If not, is it a bug? I couldn't find a
Jira but perhaps it is related to the following Jira:
https://issues.apache.org/jira/browse/HIVE-1631 ?

Thanks in advance,
Matt

[jira] [Commented] (HIVE-5412) HivePreparedStatement.setDate not implemented

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276420#comment-14276420
 ] 

Matt Burgess commented on HIVE-5412:


What about:

public void setDate(int parameterIndex, Date x, Calendar cal) throws 
SQLException {
// TODO Auto-generated method stub
throw new SQLException(Method not supported);
  }

 HivePreparedStatement.setDate not implemented
 -

 Key: HIVE-5412
 URL: https://issues.apache.org/jira/browse/HIVE-5412
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 0.12.0
Reporter: Alan Gates
 Fix For: 0.13.0


 The DATE type was added in Hive 0.12, but the HivePreparedStatement.setDate 
 method was not implemented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-4806) Add more implementations of JDBC API methods to Hive and Hive2 drivers

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13717959#comment-13717959
 ] 

Matt Burgess commented on HIVE-4806:


Alan,

Thanks for the feedback! I will incorporate them and submit a new patch, here 
are a few notes:

- HiveConnection.setReadOnly - Oops! I blindly applied my Hive 1 changes across 
to Hive 2. If HS2 supports both modes, I will update the patch to maintain the 
appropriate state.

- HiveDatabaseMetaData.supportsResultSetConcurrency - I have not tested this 
but I will try, do you know of existing test scripts that would educe this 
behavior? Alternatively, is it more appropriate to return false for Hive / HS2?

- HiveResultsSetMetaData.getColumnTypeName - I'm not really sure about these, I 
inherited a bit of this code. I did find references in MetaStoreUtils: 
https://github.com/apache/hive/blob/075e02ea6de25958de2820d73b7865e7f9d56189/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java

but I'm not sure this is germane to HiveResultsSetMetaData. I can surely remove 
these if appropriate to do so.


 Add more implementations of JDBC API methods to Hive and Hive2 drivers
 --

 Key: HIVE-4806
 URL: https://issues.apache.org/jira/browse/HIVE-4806
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Affects Versions: 0.11.0
Reporter: Matt Burgess
 Fix For: 0.12.0

 Attachments: HIVE-4806.patch


 Third-party client software such as Pentaho Data Integration (PDI) uses many 
 different JDBC API calls when interacting with JDBC data sources. Several of 
 these calls have not yet been implemented in the Hive and Hive 2 drivers and 
 by default will throw Method not supported SQLExceptions when there could 
 be default implementations instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4806) Add more implementations of JDBC API methods to Hive and Hive2 drivers

[Matt Burgess (JIRA)]

Matt Burgess created HIVE-4806:
--

 Summary: Add more implementations of JDBC API methods to Hive and 
Hive2 drivers
 Key: HIVE-4806
 URL: https://issues.apache.org/jira/browse/HIVE-4806
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Affects Versions: 0.11.0
Reporter: Matt Burgess
 Fix For: 0.12.0


Third-party client software such as Pentaho Data Integration (PDI) uses many 
different JDBC API calls when interacting with JDBC data sources. Several of 
these calls have not yet been implemented in the Hive and Hive 2 drivers and by 
default will throw Method not supported SQLExceptions when there could be 
default implementations instead.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4806) Add more implementations of JDBC API methods to Hive and Hive2 drivers

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated HIVE-4806:
---

Status: Patch Available  (was: Open)

 Add more implementations of JDBC API methods to Hive and Hive2 drivers
 --

 Key: HIVE-4806
 URL: https://issues.apache.org/jira/browse/HIVE-4806
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Affects Versions: 0.11.0
Reporter: Matt Burgess
 Fix For: 0.12.0

 Attachments: HIVE-4806.patch


 Third-party client software such as Pentaho Data Integration (PDI) uses many 
 different JDBC API calls when interacting with JDBC data sources. Several of 
 these calls have not yet been implemented in the Hive and Hive 2 drivers and 
 by default will throw Method not supported SQLExceptions when there could 
 be default implementations instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4806) Add more implementations of JDBC API methods to Hive and Hive2 drivers

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated HIVE-4806:
---

Attachment: HIVE-4806.patch

Attached proposed patch file

 Add more implementations of JDBC API methods to Hive and Hive2 drivers
 --

 Key: HIVE-4806
 URL: https://issues.apache.org/jira/browse/HIVE-4806
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Affects Versions: 0.11.0
Reporter: Matt Burgess
 Fix For: 0.12.0

 Attachments: HIVE-4806.patch


 Third-party client software such as Pentaho Data Integration (PDI) uses many 
 different JDBC API calls when interacting with JDBC data sources. Several of 
 these calls have not yet been implemented in the Hive and Hive 2 drivers and 
 by default will throw Method not supported SQLExceptions when there could 
 be default implementations instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4806) Add more implementations of JDBC API methods to Hive and Hive2 drivers

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698242#comment-13698242
 ] 

Matt Burgess commented on HIVE-4806:


Issued pull request: https://github.com/apache/hive/pull/9

 Add more implementations of JDBC API methods to Hive and Hive2 drivers
 --

 Key: HIVE-4806
 URL: https://issues.apache.org/jira/browse/HIVE-4806
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Affects Versions: 0.11.0
Reporter: Matt Burgess
 Fix For: 0.12.0

 Attachments: HIVE-4806.patch


 Third-party client software such as Pentaho Data Integration (PDI) uses many 
 different JDBC API calls when interacting with JDBC data sources. Several of 
 these calls have not yet been implemented in the Hive and Hive 2 drivers and 
 by default will throw Method not supported SQLExceptions when there could 
 be default implementations instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira