[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108602#comment-16108602
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user asfgit closed the pull request at:
https://github.com/apache/bahir/pull/48
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107340#comment-16107340
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user bchen-talend commented on the issue:
https://github.com/apache/bahir/pull/48
Thanks for your PR @ire7715, LGTM!
would you mind to remove this dependency, as it no use anymore.
```
com.google.cloud.bigdataoss
util-hadoop
1.6.0-hadoop2
```
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106791#comment-16106791
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/97/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106784#comment-16106784
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build failed, see build log for details
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106780#comment-16106780
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ire7715 commented on the issue:
https://github.com/apache/bahir/pull/48
@ckadner Thanks for suggestion!
Just added the paragraph to instruct how to generate a proper service
account and set environment variables for test cases.
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106004#comment-16106004
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/48
@ire7715 -- I create a [Google API Service
account](https://console.developers.google.com/iam-admin/serviceaccounts/project?project=apache-bahir-pubsub)
and [added the generated key
files](https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs)
to our Jenkins server. All your tests appear to be [enabled and complete
successfully](http://169.45.79.58:8080/job/bahir_spark_pr_builder/95/) now.
```
[INFO] --- scalatest-maven-plugin:1.0:test (test) @
spark-streaming-pubsub_2.11 ---
Discovery starting.
Google Pub/Sub tests that actually send data has been enabled by setting
the environment
variable ENABLE_PUBSUB_TESTS to 1.
This will create Pub/Sub Topics and Subscriptions in Google cloud platform.
Please be aware that this may incur some Google cloud costs.
Set the environment variable GCP_TEST_PROJECT_ID to the desired project.
Discovery completed in 135 milliseconds.
Run starting. Expected test count is: 10
SparkGCPCredentialsBuilderSuite:
- should build application default
- should build json service account
- should provide json creds
- should build p12 service account
- should provide p12 creds
- should build metadata service account
- SparkGCPCredentials classes should be serializable
Using project apache-bahir-pubsub for creating Pub/Sub topic and
subscription for tests.
PubsubStreamSuite:
- PubsubUtils API
- pubsub input stream
- pubsub input stream, create pubsub
Run completed in 14 seconds, 143 milliseconds.
Total number of tests run: 10
Suites: completed 3, aborted 0
Tests: succeeded 10, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
---
Would you **please add a short paragraph** to the [PubSub
README](https://github.com/apache/bahir/blob/master/streaming-pubsub/README.md)
describing how to enable your unit tests by setting the environment variables
(and how to set up a Google API *service account*, generate *key files* and how
to minimally configure the *Roles* like "Pub/Sub Publisher", etc)? i.e.:
```Bash
mvn clean package -DskipTests -pl streaming-pubsub
export ENABLE_PUBSUB_TESTS=1
export
GCP_TEST_ACCOUNT="apache-bahir-streaming-pub...@apache-bahir-pubsub.iam.gserviceaccount.com"
export GCP_TEST_PROJECT_ID="apache-bahir-pubsub"
export
GCP_TEST_JSON_KEY_PATH=/path/to/pubsub/credential/files/Apache-Bahir-PubSub-1234abcd.json
export
GCP_TEST_P12_KEY_PATH=/path/to/pubsub/credential/files/Apache-Bahir-PubSub-5678efgh.p12
mvn test -pl streaming-pubsub
```
**Thank you!**
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105996#comment-16105996
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/95/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105995#comment-16105995
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build successful
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102636#comment-16102636
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build successful
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102637#comment-16102637
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/94/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101223#comment-16101223
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/48
thanks @ire7715 -- I have a few remarks regarding your latest comment:
---
> Don't know if the force push would bother you when reviewing
Thanks for not force-pushing :+1: -- It's preferable to have multiple
commits in response to PR review comments and change requests. This makes it
much easier to come back later to see how code changes came about. Bahir
committers will squash all commits when merging Pull Requests.
So, please push another "normal" commit with your latest changes.
---
> `SparkGCPCredentialsBuilderSuite` ... ignores the test cases if the key
files or email account [environment variables] are not set (or file doesn't
exist) and shows the hint message
I agree mostly. We should ignore the test cases if env variables are not
set. However, if the environment variables **are set** and the key file **path
is invalid** then that should be an **error**. Otherwise we may not catch
problems if there are changes in the Jenkins CI server.
Could you generate a set of (permanent) key files which we can integrate
into our Jenkins PR builder?
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099522#comment-16099522
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build successful
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099523#comment-16099523
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/93/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099473#comment-16099473
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ire7715 commented on the issue:
https://github.com/apache/bahir/pull/48
Thanks @ckadner for suggestion.
Agree with your point, just modified the `SparkGCPCredentialsBuilderSuite`
and make it read the environment variables for keyfiles defined in
`PubsubTestUtils`. It ignores the test cases if the keyfiles or email account
are not set(or file doesn't exist) and shows the hint message.
Also discovered that `ServiceAccountCredentials.getFileBuffer` didn't check
file existence before opening, just added the check.
Don't know if the force push would bother you when reviewing, so I made
some manual-diff:
SparkGCPCredentials.scala L66
\- `if (filePath.isEmpty) Array[Byte]()`
\+ `if (filePath.isEmpty || !Files.exists(Paths.get(filePath.get)))
Array[Byte]()`
SparkGCPCredentialsBuilderSuite.scala
> Deleted resource variables, let `jsonFilePath`, `p12FilePath` and
`emailAccount` read system environment.
> Implemented `jsonAssumption` and `p12Assumption`, to check the existence
of corresponding environment variables and files.
> No longer expected `credential.refreshToken` thrown an Exception,
expected it could retrieve the real token.
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099461#comment-16099461
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/92/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099460#comment-16099460
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build successful
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099444#comment-16099444
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build successful
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099445#comment-16099445
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/91/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099241#comment-16099241
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/48
Thanks @ire7715 for your fixes.
Re: key file ([comment, July
21](https://github.com/apache/bahir/pull/48#discussion_r128909173))
> **ckadner:** are there no risks with making this key-file public?
> **ire7715 :** Yes, it is okay. The key was generated as a dummy IAM
service account, which now have been removed. And I have interchanged part of
the private key bytes, which makes it unusable.
So, the key file is unusable for the unit test runs? If so, then there
would be no reason to adding it as a test resource, no?
Is the idea then to communicate to developers how/where to add a key file
they would have to generate for themselves? Would it be better then to have the
unit test display a warning message in the console output if the key file is
missing and skip the impacted test case(s)?
For the Jenkins CI server, we would have to install a key file that does
work (keeps working) and in a pre-build step copy it from somewhere, or use an
environment variable to point to a local directory that has the key file.
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097501#comment-16097501
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build successful
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097495#comment-16097495
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ire7715 commented on the issue:
https://github.com/apache/bahir/pull/48
Very new to the open source licensing, didn't know RAT. Glad that I know
it now.
I have made all key files ends with `.key` and let RAT exclude them.
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097493#comment-16097493
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build failed, see build log for details
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097494#comment-16097494
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/84/
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097490#comment-16097490
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ire7715 commented on a diff in the pull request:
https://github.com/apache/bahir/pull/48#discussion_r128909248
--- Diff:
streaming-pubsub/src/main/scala/org/apache/spark/streaming/pubsub/SparkGCPCredentials.scala
---
@@ -17,10 +17,13 @@
package org.apache.spark.streaming.pubsub
+import com.google.api.client.json.jackson.JacksonFactory
--- End diff --
Ahhh, sorry. I have added
`com.google.http-client:google-http-client-jackson:1.22.0` into pom.xml.
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097488#comment-16097488
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build failed, see build log for details
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096876#comment-16096876
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on a diff in the pull request:
https://github.com/apache/bahir/pull/48#discussion_r128862044
--- Diff:
streaming-pubsub/src/main/scala/org/apache/spark/streaming/pubsub/SparkGCPCredentials.scala
---
@@ -17,10 +17,13 @@
package org.apache.spark.streaming.pubsub
+import com.google.api.client.json.jackson.JacksonFactory
--- End diff --
this dependency may have to be added to `streaming-pubsub/pom.xml`
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096877#comment-16096877
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/48
@ire7715
```
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
spark-streaming-pubsub_2.11 ---
...
[INFO] Compiling 3 Scala sources to
/var/lib/jenkins/workspace/bahir_spark_pr_builder/streaming-pubsub/target/scala-2.11/classes...
[ERROR]
/var/lib/jenkins/workspace/bahir_spark_pr_builder/streaming-pubsub/src/main/scala/org/apache/spark/streaming/pubsub/SparkGCPCredentials.scala:20:
object jackson is not a member of package com.google.api.client.json
[ERROR] import com.google.api.client.json.jackson.JacksonFactory
[ERROR] ^
[ERROR]
/var/lib/jenkins/workspace/bahir_spark_pr_builder/streaming-pubsub/src/main/scala/org/apache/spark/streaming/pubsub/SparkGCPCredentials.scala:74:
not found: type JacksonFactory
[ERROR] val jsonFactory = new JacksonFactory
[ERROR] ^
[ERROR] two errors found
```
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096867#comment-16096867
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Build failed, see build log for details
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096863#comment-16096863
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on a diff in the pull request:
https://github.com/apache/bahir/pull/48#discussion_r128861101
--- Diff:
streaming-pubsub/src/test/resources/org/apache/spark/streaming/pubusb/key-file.json
---
@@ -0,0 +1,12 @@
+{
+ "type": "service_account",
+ "project_id": "apache-bahir-streaming-pubusb",
+ "private_key_id": "**this-is-fake-key-id***",
+ "private_key": "-BEGIN PRIVATE
KEY-\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQD6c9MDG3gq3d+3\nV6AqayUNWlC/T5Qrd3YJOItNgDxZ0bAl9FakePrivateKey1dX44uR4FomugRX3s\nENwGRcEndczGcGivTfFEB8ZeEokBQWfuWoQkJXSPaJ1rYca3l//caWxBJ+DqBw67\nF9vJqyJ23Z/kFtQOdB3+5AwfJ0b8Jq5mkQF9FL6843mHjep2LhVTcKbjJBz0K+cS\nUDr4MEoxsc0jvIDf3EwbeGWPayRzB6d558eVa+OrcCKpTxGvBJmhzsI2Ol2EcypA\nIDOFZ7OkobWdxDYhM9vUCPUNKmMs0doR9Hola8XO92D2Y4q9BoCuU+hoDPEVVQOd\nOlKCuernAgMBAAECggEAe1/rJrC1dYhu2EZWJA875WQEOvncp7zlbI1qMfdlw2lE\nOK5gmcF3zIbhuKefsH38e7zVSTlFg2I4Mb3sZTqfd+zTvz1IlHL00upxkY3X58Js\njEISriu1S5/hTDCST4aVB+L27PHUHfT0EL4kCyg+hgeO6DFGrQgObq2wOviCQ1th\nPZGccIrvAXMwGA+6OaUpnPpBbXnZKarYTGLGjoVD2eLPx+viLRKl2AW9PChdkk/0\nZvHeL7bxbYHyktK8Vp4gHStBV421HkRNlvt5S31ju75P9ReHsxCpLt1OnhBHa/gD\nimlm5fWrSHoFHx2Q+zYVt/BhmWH/Dzq1Rd+e5/vkwQKBgQD9fj6SpOYnrgNLXa25\ny4p0VHXAHweH3fpqfsSJLLuc0TWDhEtrDhVTmX35N1J5J7GIWxMGFiQxlkj+6vm6\ncOfLSUYO++HOhWdIvNzRBUJ1NSa5oIfJITAH9vPYvmrdmr5+CNZAM1KsmV7CRhvJ\nScMTVjV0gqSFKEr3QKyCLwh3PwKBgQD87eHAwZp34DNNYWqPTb2Um9xegWnT5KYh\ntX3nxPRzyGfpPYeGedjWOwb5ST1KT0HlNhAPev02J6ZUhTrMjwHCnZcUlNiqDWdT\nlACNO810B98fO7GejjTEa6MqfaMG2m4UDA93hDBeuCOhHzXVfXvxLpUx0ABJR5Tg\nTMhkQ+AKWQKBgHYAysghEzLtgoMW/MQ8yBsXJillSHArGWNx17OzqzJ5AVxTvXf8\nelkMXuQgqLfVjoNXQifXLsoWl6xzXgU4ge7UEVTwVFF7MHVf1btHo4REVd6bqBos\n5NsQTrtbCQxX+M1a98GzIo1OaBov4Md3GuRpgUDXgBashxlKdgO0OVCpAoGANra6\n7Di1UpNEZcvaAk/938TroeH646SFr6sUJmv7uYQzvkfaJmP7XTR9qLWINaf5iDzu\nsnqXhfyDxargclnJNrFiekhMqlSl8nWEvQifxCbjxFzkank2vvrN3CY7ewMLZvjI\n68FFuem5g2Q+AAXaJu09xv3I4hFDClZxzkeY01kCgYEAv9a4vgpvGMHnjMEfq3Ym\ncbQIFq1l3djh4YqOy92EM0xr3nb1DEIvMshfhby5rwhejZ8j8m/lt/5t6uHd90/y\n60UcuPgJa2MgnPIIOZyQGH3C88o25WF9yvUAItbUtl9fxgJYdi/d9Hj821sZbhmF\nyZltoUeUMYMS4QW2OM6Dydk=\n-END
PRIVATE KEY-\n",
+ "client_email":
"pubsub-subscri...@apache-bahir-streaming-pubsub.iam.gserviceaccount.com",
--- End diff --
are there no risks with making this `key-file` public?
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096858#comment-16096858
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/48
@bchen-talend -- can you take a look at this PR?
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096850#comment-16096850
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/48
ok to test
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087006#comment-16087006
]
Ire Sun commented on BAHIR-122:
---
I have implemented a solution for this, here is the pull
request(https://github.com/apache/bahir/pull/48)
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable
[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087002#comment-16087002
]
ASF GitHub Bot commented on BAHIR-122:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/48
Can one of the admins verify this patch?
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
>Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
