justinleet opened a new pull request #839: HIVE-21894: Hadoop credential 
password storage for the Kafka Storage handler when security is SSL
URL: https://github.com/apache/hive/pull/839
   Allows for the KafkaStorageHandler to be configured with SSL properties, 
where the passwords aren't in plaintext in the table configs.  
   This has been tested on an actual Hadoop cluster against an actual Kafka 
cluster, but in a pretty limited manner and primarily for the consumer side of 
things (full disclosure, my use case is pretty exclusively read from).  I've 
done some basic testing to make sure both queries that aren't spinning up jobs 
(e.g. simple `SELECT *` type queries) and queries that do spin up jobs (e.g. 
some basic `GROUP BY`) all runs to success.
   There's a couple things that probably need some feedback and possibly 
   - Distribution of the key/trust stores. Kafka can only work with these 
stores locally, but they need to be distributed for jobs, so HDFS seems like 
the right place to keep them. Right now, it's an HDFS file that is being pulled 
via the standard HDFS APIs into `DOWNLOADED_RESOURCES_DIR`.  There are other 
StorageHandlers (see: 
 do some dealing with files, but they seem to do with jars and go through the 
`tmpjars` config (which I believe is just essentially `-libjars`).
     - Is this the right place to put the files?
     - Is this a more reasonable way to get them?
   - Right now, producer / consumer SSL configs are assumed to be the same 
(i.e. `hive.kafka.ssl.keystore.password` instead of `hive.kafka.consumer.ssl 
...` and `hive.kafka.producer.ssl ...`
     - This could fairly easily be split out if there's a need. I'm not 
honestly sure how much configuring a producer and consumer separately would be 
used in practice.
   - Naming of the configs. If there are any particular conventions I should 
follow, let me know and I'll test and update.
   - Automated testing. Given the need for HDFS and Kafka, I've just added some 
tests that the configs end up reasonable, but we may want more and I'm not 
familiar enough with Hive's testing utilities to know if there are better 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org

Reply via email to