ASF GitHub Bot commented on BAHIR-122:

Github user ire7715 commented on the issue:

    Thanks @ckadner for suggestion.
    Agree with your point, just modified the `SparkGCPCredentialsBuilderSuite` 
and make it read the environment variables for keyfiles defined in 
`PubsubTestUtils`. It ignores the test cases if the keyfiles or email account 
are not set(or file doesn't exist) and shows the hint message.
    Also discovered that `ServiceAccountCredentials.getFileBuffer` didn't check 
file existence before opening, just added the check.
    Don't know if the force push would bother you when reviewing, so I made 
some manual-diff:
    SparkGCPCredentials.scala L66
    \- `if (filePath.isEmpty) Array[Byte]()`
    \+ `if (filePath.isEmpty || !Files.exists(Paths.get(filePath.get))) 
    > Deleted resource variables, let `jsonFilePath`, `p12FilePath` and 
`emailAccount` read system environment.
    > Implemented `jsonAssumption` and `p12Assumption`, to check the existence 
of corresponding environment variables and files.
    > No longer expected `credential.refreshToken` thrown an Exception, 
expected it could retrieve the real token.

> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --------------------------------------------------------------
>                 Key: BAHIR-122
>                 URL: https://issues.apache.org/jira/browse/BAHIR-122
>             Project: Bahir
>          Issue Type: Improvement
>          Components: Spark Streaming Connectors
>            Reporter: Ire Sun
> The origin implementation broadcast the key file path to Spark cluster, then 
> the executor read key file with the broadcasted path. Which is absurd, if you 
> are using a shared Spark cluster in a group/company, you certainly not want 
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote 
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to 
> pull messages - java.io.FileNotFoundException}}

This message was sent by Atlassian JIRA

Reply via email to