ASF GitHub Bot commented on BAHIR-137:

Github user emlaver commented on a diff in the pull request:

    --- Diff: 
    @@ -72,6 +72,7 @@ class ChangesReceiver(config: CloudantChangesConfig)
             val status = headers.getOrElse("Status", IndexedSeq.empty)
             val errorMsg = "Error retrieving _changes feed " + 
config.getDbname + ": " + status(0)
             reportError(errorMsg, new CloudantException(errorMsg))
    +        stop(errorMsg)
    --- End diff --
    Yes, when testing with `spark-submit` command you would see two additional 
error messages.  First message below is from `reportError`, and the other two 
are from `stop`:
    18/01/04 13:04:11 WARN ReceiverSupervisorImpl: Reported error Error 
retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized - 
org.apache.bahir.cloudant.common.CloudantException: Error retrieving _changes 
feed animaldb: HTTP/1.1 401 Unauthorized
    18/01/04 13:04:11 INFO ReceiverSupervisorImpl: Stopping receiver with 
message: Error retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized: 
    18/01/04 13:04:11 ERROR ReceiverTracker: Deregistered receiver for stream 
0: Error retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized
    Since we already call `stop` [in 
 I'll remove this line.
    I will also be changing [this error 
 as it's more likely an HTTP error happened.  For users running with 
`spark-submit`, they will get logging in the console.  There will be limited or 
no logging in DSX which means it might be worth passing the HTTP error message 
to `CloudantException`.

> Load performance improvements for _changes API in sql-cloudant
> --------------------------------------------------------------
>                 Key: BAHIR-137
>                 URL: https://issues.apache.org/jira/browse/BAHIR-137
>             Project: Bahir
>          Issue Type: Improvement
>    Affects Versions: Spark-2.2.0
>            Reporter: Esteban Laver
>            Assignee: Esteban Laver
> Items for improving _changes feed load:
> - Make Spark streaming batch interval visible to the user for tuning based on 
> type/size of document and number of docs in database
> - Merge BAHIR-128: Improve stability of _changes receiver
> - Merge BAHIR-154: refactor sql-cloudant to use java-cloudant library

This message was sent by Atlassian JIRA

Reply via email to