[
https://issues.apache.org/jira/browse/SAMZA-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892865#comment-15892865
]
ASF GitHub Bot commented on SAMZA-1085:
---------------------------------------
GitHub user fredji97 opened a pull request:
https://github.com/apache/samza/pull/72
SAMZA-1085 add CertFSFileSystem and SimpleCertFSFileSystem for handling
certificate localization from CSR in yarn
This is the implementation used during the experiment and testing. will add
more unit tests and improve the TODO section in later PRs.
Currently implementation is tested manually with the following yarn
configuration (yarn core-site.xml):
<property>
<name>fs.certfs.impl</name>
<value>org.apache.samza.util.hadoop.CertFSFileSystem</value>
</property>
<property>
<name>csr.ssl.truststore.type</name>
<value>${csr.ssl.truststore.type}</value>
</property>
<property>
<name>csr.ssl.truststore.location</name>
<value>${csr.ssl.truststore.location}</value>
</property>
<property>
<name>csr.ssl.truststore.password</name>
<value>${csr.ssl.truststore.password}</value>
</property>
<property>
<name>csr.ssl.keystore.type</name>
<value>${csr.ssl.keystore.type}</value>
</property>
<property>
<name>csr.ssl.keystore.location</name>
<value>${csr.ssl.keystore.location}</value>
</property>
<property>
<name>csr.ssl.keystore.password</name>
<value>${csr.ssl.keystore.password}</value>
</property>
<property>
<name>csr.ssl.key.password</name>
<value>${csr.ssl.key.password}</value>
</property>
<property>
<name>csr.ssl.bcstyle.x500name</name>
<value>${csr.ssl.bcstyle.x500name}</value>
</property>
The tested uri is
certfs://lca1-app0739.corp.linkedin.com:10180/grestin/request_service_certificate?fabric=dev&application=test_application&instance=i001
The generated certificate is found in the corresponding application cache
for the specific application. (in my testing env, it is
export/content/data/samsa-yarn/usercache/samza-ji/ap461_0093/container_e03_1487135111461_0093_02_000001/identity.p12)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fredji97/samza certfs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/72.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #72
----
commit d46b5371e8a3c2001d5d346d93617a921ab8fd1f
Author: Fred Ji <[email protected]>
Date: 2017-03-02T18:24:06Z
SAMZA-1085: adding the samza-yarn dependency on httpClient and
bouncyCastle, which are used for https post request and json parsing on CSR
request/response
commit f1a196d1b25d0192bf086c71e72d683ce2cae674
Author: Fred Ji <[email protected]>
Date: 2017-03-02T19:41:28Z
SAMZA-1085 add CertFSFileSystem and SimpleCertFSFileSystem for handling
certificate localization from CSR in yarn
----
> add a new filesystem for localizing the certificate from CSR in yarn
> --------------------------------------------------------------------
>
> Key: SAMZA-1085
> URL: https://issues.apache.org/jira/browse/SAMZA-1085
> Project: Samza
> Issue Type: New Feature
> Reporter: Fred Ji
> Assignee: Fred Ji
>
> Currently, for the samza jobs running in yarn, there is no support for ACL
> when accessing the dependent service such as Kafka, or any restful service.
> In order to protect the data and isolate the data among the samza jobs, we
> may need to have ACL enabled for these services. For https services
> supporting TLS/SSL, one of the requirements is to have the key store which
> includes the client side certificates for these Samza jobs so when they call
> the service, services know who they are and whether they have access or not.
> When Samza runs on cluster based system such as YARN, the certificate
> distribution could be very challenging because the NM servers do not have app
> specific certificate originally. One of common ways is to have CA which sign
> the certificates for the Samza jobs and then Samza jobs keep these
> certificates in the localized directory and use them for https communication.
> The process of requesting the app specific certificate is called CSR. It is
> mostly a https request to CA, but besides that, it also needs to generate the
> public key and private key pair, and the certificate string needs to put
> together with private key in the local directory for the later app level
> https communication.
> Considering the complicated process of CSR, we are considering create a new
> scheme called certfs, and have a config called "fs.certfs.impl" which map to
> a class for localization. The default class will be CertFSFileSystem, but the
> user can implement other specific class for fs.certfs.impl.
> CertFSFileSystem extends org.apache.hadoop.fs.FileSystem, and majorly do the
> following:
> 1. get the original certfs:// uri;
> 2. reconstruct the https:// uri for csr from certfs://
> 3. generate public/private key pair
> 4. prepare the csr request (ssl context including keystore, truststore, and
> https csr payload)
> 5. send request and fetch the signed certificate
> 6. combine the signed certificate and private key, and localize the key store
> for this specific app
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
