Hi all. I’m attempting to add SSL support to the Kafka storage handler and will need to make the keystore/truststore available to KafkaConsumer via an absolute path on a local file system.
The ideal steps are these: 1. TBLPROPERTIES describe an absolute HDFS path for the keystore/truststore 2. The Kafka storage handler copies both files from HDFS to the container’s local FS and configures and builds the KafkaConsumer around an absolute reference to this local YARN container directory. 3. Passwords for these files are stored using TBLPROPERTIES references to Hadoop credentials similar to the examples in the JDBC storage handler Looks like there is precedent for interacting with HDFS from a storage handler level for HBase here: https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java#L366, then in the Druid Storage handler there is a reference to using the local filesystem for dependency jar additions: https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java#L692 Need some help with a few questions: 1. Is that the FS under the temporary YARN container example in the second link where all software/configuration is distributed? 2. Is this Storage handler execution scope at the ‘mapper’ level on a container running YARN? 3. Do these file movement steps seem ok and within a storage handler’s scope? Thanks, Kris
