Fred Ji created SAMZA-1085:
------------------------------
Summary: add a new filesystem for localizing the certificate from
CSR in yarn
Key: SAMZA-1085
URL: https://issues.apache.org/jira/browse/SAMZA-1085
Project: Samza
Issue Type: New Feature
Reporter: Fred Ji
Assignee: Fred Ji
Currently, for the samza jobs running in yarn, there is no support for ACL when
accessing the dependent service such as Kafka, or any restful service. In order
to protect the data and isolate the data among the samza jobs, we may need to
have ACL enabled for these services. One of the requirements is to have the
client side certificates for these Samza jobs so when they call the service,
services know who they are and whether they have access or not.
When Samza runs on cluster based system such as YARN, the certificate
distribution could be very challenging because the NM servers do not have app
specific certificate originally. One of a common way is to have CA which sign
the certificates for the Samza jobs and then Samza jobs keep these certificates
in the localized directory and use them for https communication.
The process of requesting the app specific certificate is called CSR. It is
mostly a https request to CA, but besides that, it also needs to generate the
public key and private key pair, and the certificate string needs to put
together with private key in the local directory for the later app level https
communication.
Considering the complicated process of CSR, we are considering create a new
scheme called certfs, and have a config called "fs.certfs.impl" which map to a
class for localization. The default class will be CertFSFileSystem, but the
user can implement other specific class for fs.certfs.impl.
CertFSFileSystem extends org.apache.hadoop.fs.FileSystem, and majorly do the
following:
1. get the original certfs:// uri;
2. reconstruct the https:// uri for csr from certfs://
3. generate public/private key pair
4. prepare the csr request (ssl context including keystore, truststore, and
https csr payload)
5. send request and fetch the signed certificate
6. combine the signed certificate and private key, and localize the key store
for this specific app
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
