ExecuteHQL failed to reading table from Hive

[Rohit]

Hi,

  I am trying read table from hive using ExecuteHQL  processor, but getting
"org.apache.avro.AvroRuntimeException: Unknown datum type java.lang.Short:
1" error.





--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: HiveConnectionPool URL with trustStorePasswrd

[Matt Burgess]

Milan,

You could subclass the existing HiveConnectionPool, add a property for
key/truststore password that is sensitive, then build the URL manually
during a call to getConnection(). Then the rest of the URL would not
have to remain sensitive, if that is preferable.  You might even
consider an SSLContext property rather than just passwords, then you
can get all that info from the SSLContext controller service and build
the URL from there.

Regards,
Matt

On Wed, Mar 14, 2018 at 3:11 PM, Milan Das  wrote:
> We are going to use a Custom controller and mark the URL as sensitive 
> property. Client don’t want to make any password visible.
>
>
>
> Regards,
>
> Milan
>
>
>
> From: Andy LoPresto 
> Reply-To: 
> Date: Monday, March 12, 2018 at 4:00 PM
> To: 
> Subject: Re: HiveConnectionPool URL with trustStorePasswrd
>
>
>
> Milan,
>
>
>
> I am also not aware of any way to use an encrypted value in the JDBC 
> connection string. In my understanding, the truststore password is only used 
> to verify the integrity of the truststore which is used locally (i.e. not 
> transmitted) to accept the remote endpoint’s TLS certificate.
>
>
>
> You could probably write a custom controller service replacing 
> HiveConnectionPool [1] which implemented HiveDBCPService and marked the 
> connection string as a sensitive property, so it would be encrypted on disk 
> by NiFi and not revealed over the API, but it might be difficult to use in 
> this way because the entire connection string would be hidden in the UI. You 
> could also theoretically have separate property descriptors for the 
> connection string and truststore password and construct the connection string 
> yourself internally, but this is probably overkill.
>
>
>
> [1] 
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-hive-nar/1.5.0/org.apache.nifi.dbcp.hive.HiveConnectionPool/index.html
>
>
>
>
>
> Andy LoPresto
>
> alopre...@apache.org
>
> alopresto.apa...@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Mar 12, 2018, at 12:27 PM, Pierre Villard  
> wrote:
>
>
>
> Hi Milan,
>
> As far as I know, there is not. It's the same when you connect with the
> beeline client from a node.
> Note that you can set the chmod/chown of the truststore file to be only
> readable by the user running nifi. It should help preventing unauthorized
> access.
>
> Pierre
>
> 2018-03-12 14:49 GMT+01:00 Milan Das :
>
>
>
> Hello folks,
>
> I am connecting to Hive environment with TLS security on. In order to do
> that need to send trustStorePasswrd  in Hive2 URL . As the configuration is
> in controller services, not able to find a way to set the
> trustStorePassword in encrypted format.
>
> Wondering if there is a way to set trustStorePassword in encrypted format ?
>
>
>
> Database ConnectionUrl: jdbc:hive2://ip-xxx-xx-x-xxx.
> ec2.internal:1/default;principal=hive/_h...@co.acme.com
> ;ssl=true;sslTrustStore=/etc/hadoop/ssl/truststore.jks;
> trustStorePassword=password
>
>
>
> Regard,
>
>
>
> [image: graph]
>
> *Milan Das*
> Sr. System Architect
>
> email: m...@interset.com
> mobile: +1 678 216 5660 <(678)%20216-5660>
>
> [image: dIn icon] 
>
> www.interset.com
>
>
>

Re: HiveConnectionPool URL with trustStorePasswrd

[Milan Das]

We are going to use a Custom controller and mark the URL as sensitive property. 
Client don’t want to make any password visible.

 

Regards,

Milan

 

From: Andy LoPresto 
Reply-To: 
Date: Monday, March 12, 2018 at 4:00 PM
To: 
Subject: Re: HiveConnectionPool URL with trustStorePasswrd

 

Milan,

 

I am also not aware of any way to use an encrypted value in the JDBC connection 
string. In my understanding, the truststore password is only used to verify the 
integrity of the truststore which is used locally (i.e. not transmitted) to 
accept the remote endpoint’s TLS certificate. 

 

You could probably write a custom controller service replacing 
HiveConnectionPool [1] which implemented HiveDBCPService and marked the 
connection string as a sensitive property, so it would be encrypted on disk by 
NiFi and not revealed over the API, but it might be difficult to use in this 
way because the entire connection string would be hidden in the UI. You could 
also theoretically have separate property descriptors for the connection string 
and truststore password and construct the connection string yourself 
internally, but this is probably overkill.  

 

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-hive-nar/1.5.0/org.apache.nifi.dbcp.hive.HiveConnectionPool/index.html

 

 

Andy LoPresto

alopre...@apache.org

alopresto.apa...@gmail.com

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

 

On Mar 12, 2018, at 12:27 PM, Pierre Villard  
wrote:

 

Hi Milan,

As far as I know, there is not. It's the same when you connect with the
beeline client from a node.
Note that you can set the chmod/chown of the truststore file to be only
readable by the user running nifi. It should help preventing unauthorized
access.

Pierre

2018-03-12 14:49 GMT+01:00 Milan Das :



Hello folks,

I am connecting to Hive environment with TLS security on. In order to do
that need to send trustStorePasswrd  in Hive2 URL . As the configuration is
in controller services, not able to find a way to set the
trustStorePassword in encrypted format.

Wondering if there is a way to set trustStorePassword in encrypted format ?



Database ConnectionUrl: jdbc:hive2://ip-xxx-xx-x-xxx.
ec2.internal:1/default;principal=hive/_h...@co.acme.com
;ssl=true;sslTrustStore=/etc/hadoop/ssl/truststore.jks;
trustStorePassword=password



Regard,



[image: graph]

*Milan Das*
Sr. System Architect

email: m...@interset.com
mobile: +1 678 216 5660 <(678)%20216-5660>

[image: dIn icon] 

www.interset.com

 

Re: Expression Language Guide

[Juan Sequeiros]

Hi Chuck,

We worked together on gmdds. How goers it.

Did you ever solve this?
This is an old thread but I had a requirement to do something similar.
Attached is the latest PDF for Expression language.
Or if you want I can give you my docker image that has the conversion tools.
It comes in handy if you ever need to provide deliverables in an isolated
environment.

On Sat, Feb 4, 2017 at 7:49 AM Chuck Dean  wrote:

> Hi folks,
>
> Is the NiFi Expression Language Guide available for PDF download? I'm one
> of those old guys who likes things on hard copy so I can highlight, circle
> and mark up.. kinda hard to do on my laptop.  ;)
>
> Thanks,
> Chuck
>