[ https://issues.apache.org/jira/browse/BAHIR-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140139#comment-16140139 ]
ASF GitHub Bot commented on BAHIR-130: -------------------------------------- Github user emlaver commented on a diff in the pull request: https://github.com/apache/bahir/pull/49#discussion_r135042956 --- Diff: sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreDataAccess.scala --- @@ -125,6 +125,19 @@ class JsonStoreDataAccess (config: CloudantConfig) { val clRequest: HttpRequest = getClRequest(url, postData) val clResponse: HttpResponse[String] = clRequest.execute() + + //BAHIR-130 start + //In cloudant Lite number of requests pre second are limited. + //Just wait and try again + clResponse.code match { + case 429 => { + Thread sleep 1000 --- End diff -- Also, would be good to have logging for: - When there's too many requests and attempting a retry with x seconds - Any problems/interrupt during the 'backoff' wait time > Support Cloudant Lite Plan > -------------------------- > > Key: BAHIR-130 > URL: https://issues.apache.org/jira/browse/BAHIR-130 > Project: Bahir > Issue Type: Improvement > Components: Spark SQL Data Sources > Affects Versions: Spark-2.0.0, Spark-2.0.1, Spark-2.0.2, Spark-2.1.0, > Spark-2.1.1, Spark-2.2.0 > Environment: ApacheSpark, any > Reporter: Romeo Kienzer > Assignee: Romeo Kienzer > Priority: Minor > Fix For: Spark-2.1.1, Spark-2.2.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Cloudant has a plan called "Lite" supporting only five requests per second. > So you end up with the following exception: > org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in > stage 0.0 failed 10 times, most recent failure: Lost task 4.9 in stage 0.0 > (TID 42, yp-spark-dal09-env5-0040): java.lang.RuntimeException: Database > harlemshake2 request error: {"error":"too_many_requests","reason":"You've > exceeded your current limit of 5 requests per second for query class. Please > try later.","class":"query","rate":5} > at > org.apache.bahir.cloudant.common.JsonStoreDataAccess.getQueryResult(JsonStoreDataAccess.scala:158) > at > org.apache.bahir.cloudant.common.JsonStoreDataAccess.getIterator(JsonStoreDataAccess.scala:72) > Suggestion: Change JsonStoreDataAccess.scala in a way that when a 403 HTTP > status code is returned the response is parsed in order to obtain the rate > limit and then throttle the query down to that limit. In addition issue a > WARNING in the log -- This message was sent by Atlassian JIRA (v6.4.14#64029)