[ 
https://issues.apache.org/jira/browse/BAHIR-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140120#comment-16140120
 ] 

ASF GitHub Bot commented on BAHIR-130:
--------------------------------------

Github user emlaver commented on a diff in the pull request:

    https://github.com/apache/bahir/pull/49#discussion_r135038265
  
    --- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreDataAccess.scala
 ---
    @@ -125,6 +125,19 @@ class JsonStoreDataAccess (config: CloudantConfig)  {
         val clRequest: HttpRequest = getClRequest(url, postData)
     
         val clResponse: HttpResponse[String] = clRequest.execute()
    +          
    +    //BAHIR-130 start
    +    //In cloudant Lite number of requests pre second are limited.
    +    //Just wait and try again
    +    clResponse.code match {
    +        case 429 => {
    +            Thread sleep 1000
    --- End diff --
    
    If the server is busy, we don't want this to loop and keep trying forever.  
We should have a set number of retries (our other Cloudant java library uses 3 
retries) and exit with a 429 error if we reach that limit.  


> Support Cloudant Lite Plan
> --------------------------
>
>                 Key: BAHIR-130
>                 URL: https://issues.apache.org/jira/browse/BAHIR-130
>             Project: Bahir
>          Issue Type: Improvement
>          Components: Spark SQL Data Sources
>    Affects Versions: Spark-2.0.0, Spark-2.0.1, Spark-2.0.2, Spark-2.1.0, 
> Spark-2.1.1, Spark-2.2.0
>         Environment: ApacheSpark, any
>            Reporter: Romeo Kienzer
>            Assignee: Romeo Kienzer
>            Priority: Minor
>             Fix For: Spark-2.1.1, Spark-2.2.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Cloudant has a plan called "Lite" supporting only five requests per second. 
> So you end up with the following exception:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in 
> stage 0.0 failed 10 times, most recent failure: Lost task 4.9 in stage 0.0 
> (TID 42, yp-spark-dal09-env5-0040): java.lang.RuntimeException: Database 
> harlemshake2 request error: {"error":"too_many_requests","reason":"You've 
> exceeded your current limit of 5 requests per second for query class. Please 
> try later.","class":"query","rate":5}
>       at 
> org.apache.bahir.cloudant.common.JsonStoreDataAccess.getQueryResult(JsonStoreDataAccess.scala:158)
>       at 
> org.apache.bahir.cloudant.common.JsonStoreDataAccess.getIterator(JsonStoreDataAccess.scala:72)
> Suggestion: Change JsonStoreDataAccess.scala in a way that when a 403 HTTP 
> status code is returned the response is parsed in order to obtain the rate 
> limit and then throttle the query down to that limit. In addition issue a 
> WARNING in the log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to