[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087001#comment-16087001
]
ASF GitHub Bot commented on BAHIR-122:
--------------------------------------
GitHub user ire7715 opened a pull request:
https://github.com/apache/bahir/pull/48
[BAHIR-122] [PubSub] Make "ServiceAccountCredentials" really broadcastable
# Make "ServiceAccountCredentials" really broadcastable
## Issue
The origin implementation broadcast the key file path to Spark cluster,
then the executor read key file with the broadcasted path. Which is absurd, if
you are using a shared Spark cluster in a group, you certainly not want to (and
have no right to) put your key file on each instance of the cluster.
## Solution
Instead of broadcasting the key file path onto the cluster, I read the key
file content in the driver node and store the binary in the
`ServiceAccountCredentials`. Whenever the provider is called, it retrieve the
credential with the in-memory key file.
The MetadataServiceAccount shall read the credential on the local instance,
since its origin purpose is for GCE instances.
## Implementation
1. Read the `BinaryArray` into `ServiceAccountCredentials.fileBytes`
2. Determine which kind of key file to use, and create Credential. (Refer
to:
[com.google.cloud.hadoop.util.CredentialConfiguration.java](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/util/src/main/java/com/google/cloud/hadoop/util/CredentialConfiguration.java#L64)
and
[com.google.cloud.hadoop.util.CredentialFactory.java](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/util/src/main/java/com/google/cloud/hadoop/util/CredentialFactory.java#L225))
## Test Case
1. Introduced two key files (.json, .p12) in package resource.
2.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ire7715/bahir
feature/20170713_Ire_broadcastable_credential
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bahir/pull/48.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #48
----
commit d42d476a35d91eddeb57760eca197383edccc419
Author: ire_sun <ire7...@hotmail.com>
Date: 2017-07-13T06:47:21Z
broadcast keyfile content so the credential provider need no keyfile
commit b666870a6b6fcf9166413a834c4b059ea491829a
Author: ire_sun <ire7...@hotmail.com>
Date: 2017-07-14T07:18:49Z
update test cases and introduce testing credential
----
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --------------------------------------------------------------
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
> Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
