[ 
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099473#comment-16099473
 ] 

ASF GitHub Bot commented on BAHIR-122:
--------------------------------------

Github user ire7715 commented on the issue:

    https://github.com/apache/bahir/pull/48
  
    Thanks @ckadner for suggestion.
    Agree with your point, just modified the `SparkGCPCredentialsBuilderSuite` 
and make it read the environment variables for keyfiles defined in 
`PubsubTestUtils`. It ignores the test cases if the keyfiles or email account 
are not set(or file doesn't exist) and shows the hint message.
    Also discovered that `ServiceAccountCredentials.getFileBuffer` didn't check 
file existence before opening, just added the check.
    
    Don't know if the force push would bother you when reviewing, so I made 
some manual-diff:
    
    ----
    
    SparkGCPCredentials.scala L66
    \- `if (filePath.isEmpty) Array[Byte]()`
    \+ `if (filePath.isEmpty || !Files.exists(Paths.get(filePath.get))) 
Array[Byte]()`
    
    ----
    
    SparkGCPCredentialsBuilderSuite.scala
    > Deleted resource variables, let `jsonFilePath`, `p12FilePath` and 
`emailAccount` read system environment.
    > Implemented `jsonAssumption` and `p12Assumption`, to check the existence 
of corresponding environment variables and files.
    > No longer expected `credential.refreshToken` thrown an Exception, 
expected it could retrieve the real token.


> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --------------------------------------------------------------
>
>                 Key: BAHIR-122
>                 URL: https://issues.apache.org/jira/browse/BAHIR-122
>             Project: Bahir
>          Issue Type: Improvement
>          Components: Spark Streaming Connectors
>            Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then 
> the executor read key file with the broadcasted path. Which is absurd, if you 
> are using a shared Spark cluster in a group/company, you certainly not want 
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote 
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to 
> pull messages - java.io.FileNotFoundException}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to