[ 
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106004#comment-16106004
 ] 

ASF GitHub Bot commented on BAHIR-122:
--------------------------------------

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/48
  
    @ire7715 -- I create a [Google API Service 
account](https://console.developers.google.com/iam-admin/serviceaccounts/project?project=apache-bahir-pubsub)
 and [added the generated key 
files](https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs)
 to our Jenkins server. All your tests appear to be [enabled and complete 
successfully](http://169.45.79.58:8080/job/bahir_spark_pr_builder/95/) now.
    
    ```
    [INFO] --- scalatest-maven-plugin:1.0:test (test) @ 
spark-streaming-pubsub_2.11 ---
    Discovery starting.
    
    Google Pub/Sub tests that actually send data has been enabled by setting 
the environment
    variable ENABLE_PUBSUB_TESTS to 1.
    This will create Pub/Sub Topics and Subscriptions in Google cloud platform.
    Please be aware that this may incur some Google cloud costs.
    Set the environment variable GCP_TEST_PROJECT_ID to the desired project.
            
    Discovery completed in 135 milliseconds.
    Run starting. Expected test count is: 10
    SparkGCPCredentialsBuilderSuite:
    - should build application default
    - should build json service account
    - should provide json creds
    - should build p12 service account
    - should provide p12 creds
    - should build metadata service account
    - SparkGCPCredentials classes should be serializable
    Using project apache-bahir-pubsub for creating Pub/Sub topic and 
subscription for tests.
    PubsubStreamSuite:
    - PubsubUtils API
    - pubsub input stream
    - pubsub input stream, create pubsub
    
    Run completed in 14 seconds, 143 milliseconds.
    Total number of tests run: 10
    Suites: completed 3, aborted 0
    Tests: succeeded 10, failed 0, canceled 0, ignored 0, pending 0
    All tests passed.
    ```
    
    ---
    
    Would you **please add a short paragraph** to the [PubSub 
README](https://github.com/apache/bahir/blob/master/streaming-pubsub/README.md) 
describing how to enable your unit tests by setting the environment variables 
(and how to set up a Google API *service account*, generate *key files* and how 
to minimally configure the *Roles* like "Pub/Sub Publisher", etc)? i.e.:
    
    ```Bash
    mvn clean package -DskipTests -pl streaming-pubsub
    
    export ENABLE_PUBSUB_TESTS=1
    export 
GCP_TEST_ACCOUNT="apache-bahir-streaming-pub...@apache-bahir-pubsub.iam.gserviceaccount.com"
    export GCP_TEST_PROJECT_ID="apache-bahir-pubsub"
    export 
GCP_TEST_JSON_KEY_PATH=/path/to/pubsub/credential/files/Apache-Bahir-PubSub-1234abcd.json
 
    export 
GCP_TEST_P12_KEY_PATH=/path/to/pubsub/credential/files/Apache-Bahir-PubSub-5678efgh.p12
    
    mvn test -pl streaming-pubsub
    ```
    
    **Thank you!**


> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --------------------------------------------------------------
>
>                 Key: BAHIR-122
>                 URL: https://issues.apache.org/jira/browse/BAHIR-122
>             Project: Bahir
>          Issue Type: Improvement
>          Components: Spark Streaming Connectors
>            Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then 
> the executor read key file with the broadcasted path. Which is absurd, if you 
> are using a shared Spark cluster in a group/company, you certainly not want 
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote 
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to 
> pull messages - java.io.FileNotFoundException}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to