[
https://issues.apache.org/jira/browse/SAMZA-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred Ji updated SAMZA-1085:
---------------------------
Description:
Currently, for the samza jobs running in yarn, there is no support for ACL when
accessing the dependent service such as Kafka, or any restful service. In order
to protect the data and isolate the data among the samza jobs, we may need to
have ACL enabled for these services. For https services supporting TLS/SSL, one
of the requirements is to have the key store which includes the client side
certificates for these Samza jobs so when they call the service, services know
who they are and whether they have access or not.
When Samza runs on cluster based system such as YARN, the certificate
distribution could be very challenging because the NM servers do not have app
specific certificate originally. One of common ways is to have CA which sign
the certificates for the Samza jobs and then Samza jobs keep these certificates
in the localized directory and use them for https communication.
The process of requesting the app specific certificate is called CSR. It is
mostly a https request to CA, but besides that, it also needs to generate the
public key and private key pair, and the certificate string needs to put
together with private key in the local directory for the later app level https
communication.
Considering the complicated process of CSR, we are considering create a new
scheme called certfs, and have a config called "fs.certfs.impl" which map to a
class for localization. The default class will be CertFSFileSystem, but the
user can implement other specific class for fs.certfs.impl.
CertFSFileSystem extends org.apache.hadoop.fs.FileSystem, and majorly do the
following:
1. get the original certfs:// uri;
2. reconstruct the https:// uri for csr from certfs://
3. generate public/private key pair
4. prepare the csr request (ssl context including keystore, truststore, and
https csr payload)
5. send request and fetch the signed certificate
6. combine the signed certificate and private key, and localize the key store
for this specific app
was:
Currently, for the samza jobs running in yarn, there is no support for ACL when
accessing the dependent service such as Kafka, or any restful service. In order
to protect the data and isolate the data among the samza jobs, we may need to
have ACL enabled for these services. One of the requirements is to have the
client side certificates for these Samza jobs so when they call the service,
services know who they are and whether they have access or not.
When Samza runs on cluster based system such as YARN, the certificate
distribution could be very challenging because the NM servers do not have app
specific certificate originally. One of a common way is to have CA which sign
the certificates for the Samza jobs and then Samza jobs keep these certificates
in the localized directory and use them for https communication.
The process of requesting the app specific certificate is called CSR. It is
mostly a https request to CA, but besides that, it also needs to generate the
public key and private key pair, and the certificate string needs to put
together with private key in the local directory for the later app level https
communication.
Considering the complicated process of CSR, we are considering create a new
scheme called certfs, and have a config called "fs.certfs.impl" which map to a
class for localization. The default class will be CertFSFileSystem, but the
user can implement other specific class for fs.certfs.impl.
CertFSFileSystem extends org.apache.hadoop.fs.FileSystem, and majorly do the
following:
1. get the original certfs:// uri;
2. reconstruct the https:// uri for csr from certfs://
3. generate public/private key pair
4. prepare the csr request (ssl context including keystore, truststore, and
https csr payload)
5. send request and fetch the signed certificate
6. combine the signed certificate and private key, and localize the key store
for this specific app
> add a new filesystem for localizing the certificate from CSR in yarn
> --------------------------------------------------------------------
>
> Key: SAMZA-1085
> URL: https://issues.apache.org/jira/browse/SAMZA-1085
> Project: Samza
> Issue Type: New Feature
> Reporter: Fred Ji
> Assignee: Fred Ji
>
> Currently, for the samza jobs running in yarn, there is no support for ACL
> when accessing the dependent service such as Kafka, or any restful service.
> In order to protect the data and isolate the data among the samza jobs, we
> may need to have ACL enabled for these services. For https services
> supporting TLS/SSL, one of the requirements is to have the key store which
> includes the client side certificates for these Samza jobs so when they call
> the service, services know who they are and whether they have access or not.
> When Samza runs on cluster based system such as YARN, the certificate
> distribution could be very challenging because the NM servers do not have app
> specific certificate originally. One of common ways is to have CA which sign
> the certificates for the Samza jobs and then Samza jobs keep these
> certificates in the localized directory and use them for https communication.
> The process of requesting the app specific certificate is called CSR. It is
> mostly a https request to CA, but besides that, it also needs to generate the
> public key and private key pair, and the certificate string needs to put
> together with private key in the local directory for the later app level
> https communication.
> Considering the complicated process of CSR, we are considering create a new
> scheme called certfs, and have a config called "fs.certfs.impl" which map to
> a class for localization. The default class will be CertFSFileSystem, but the
> user can implement other specific class for fs.certfs.impl.
> CertFSFileSystem extends org.apache.hadoop.fs.FileSystem, and majorly do the
> following:
> 1. get the original certfs:// uri;
> 2. reconstruct the https:// uri for csr from certfs://
> 3. generate public/private key pair
> 4. prepare the csr request (ssl context including keystore, truststore, and
> https csr payload)
> 5. send request and fetch the signed certificate
> 6. combine the signed certificate and private key, and localize the key store
> for this specific app
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
