[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16312523#comment-16312523
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159814009
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
--- End diff --

This is a great idea and I'll use this for both view and search.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-137) Load performance improvements for _changes API in sql-cloudant

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311810#comment-16311810
 ] 

ASF GitHub Bot commented on BAHIR-137:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/60#discussion_r159722257
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/internal/ChangesReceiver.scala
 ---
@@ -72,6 +72,7 @@ class ChangesReceiver(config: CloudantChangesConfig)
 val status = headers.getOrElse("Status", IndexedSeq.empty)
 val errorMsg = "Error retrieving _changes feed " + 
config.getDbname + ": " + status(0)
 reportError(errorMsg, new CloudantException(errorMsg))
+stop(errorMsg)
--- End diff --

Yes, when testing with `spark-submit` command you would see two additional 
error messages.  First message below is from `reportError`, and the other two 
are from `stop`:
```
...
18/01/04 13:04:11 WARN ReceiverSupervisorImpl: Reported error Error 
retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized - 
org.apache.bahir.cloudant.common.CloudantException: Error retrieving _changes 
feed animaldb: HTTP/1.1 401 Unauthorized
...
18/01/04 13:04:11 INFO ReceiverSupervisorImpl: Stopping receiver with 
message: Error retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized: 
...
18/01/04 13:04:11 ERROR ReceiverTracker: Deregistered receiver for stream 
0: Error retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized

```
Since we already call `stop` [in 
DefaultSource](https://github.com/apache/bahir/pull/60/files#diff-4a0eb687595fe14565677d020371b69eR154),
 I'll remove this line.
I will also be changing [this error 
message](https://github.com/apache/bahir/pull/60/files#diff-4a0eb687595fe14565677d020371b69eR165)
 as it's more likely an HTTP error happened.  For users running with 
`spark-submit`, they will get logging in the console.  There will be limited or 
no logging in DSX which means it might be worth passing the HTTP error message 
to `CloudantException`.


> Load performance improvements for _changes API in sql-cloudant
> --
>
> Key: BAHIR-137
> URL: https://issues.apache.org/jira/browse/BAHIR-137
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Items for improving _changes feed load:
> - Make Spark streaming batch interval visible to the user for tuning based on 
> type/size of document and number of docs in database
> - Merge BAHIR-128: Improve stability of _changes receiver
> - Merge BAHIR-154: refactor sql-cloudant to use java-cloudant library



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311722#comment-16311722
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159699042
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
--- End diff --

I think you could leverage scala's raw interpreter here to make the regex a 
bit less escape-y:
`raw"\w+\/\w+\/\w+\/\w+"`

Also are word characters sufficient here? Aren't there some other 
characters that could be part of a design document id and allowed unencoded in 
a URL path (e.g. maybe `@`?) that would be excluded by this regex? I'd be 
inclined to maybe use something like: `_design\/(^\/)+\/_search\/(^\/)+`


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311723#comment-16311723
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159699798
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
--- End diff --

If you separate out the regex pattern from earlier you could use the 
captured groups to extract the design doc ID and search index name without 
needing to do more splits here.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311721#comment-16311721
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159709935
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "&skip=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
+}
+  }
+
+  def getDocs(limit: Int): List[JsonObject] = {
+if (viewPath != null) {
+  // "limit=" + limit + "&skip=" + skip
+  
buildViewRequest(limit).build().getResponse.getDocsAs(classOf[JsonObject]).asScala.toList
+} else if (indexPath != null) {
+  var searchDocs = mutable.ListBuffer[JsonObject]()
+  for (result: SearchResult[JsonObject]#SearchResultRow <-
+   buildSearchRequest(limit).getRows.asScala) {
+searchDocs += result.getDoc
+  }
+  searchDocs.toList
+} else {
+  // /_all_docs?limit=1
+  // val response = client.executeRequest(Http.GET(
--- End diff --

Remove commented out code?


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311720#comment-16311720
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159709382
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "&skip=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
--- End diff --

It might be easier to use
`com.cloudant.client.api.model.DbInfo#getDocCount` instead of trying to do 
it via `_all_docs`


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-137) Load performance improvements for _changes API in sql-cloudant

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311425#comment-16311425
 ] 

ASF GitHub Bot commented on BAHIR-137:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/60#discussion_r159669507
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreConfigManager.scala
 ---
@@ -35,6 +35,7 @@ object JsonStoreConfigManager {
   private val CLOUDANT_PASSWORD_CONFIG = "cloudant.password"
   private val CLOUDANT_PROTOCOL_CONFIG = "cloudant.protocol"
   private val CLOUDANT_API_ENDPOINT = "cloudant.endpoint"
+  private val CLOUDANT_STREAMING_BATCH_INTERVAL = "cloudant.batchInterval"
   private val STORAGE_LEVEL_FOR_CHANGES_INDEX = "cloudant.storageLevel"
--- End diff --

Yes, they should be the same.  I'll revert the documentation back to 
`cloudant.storageLevel`.


> Load performance improvements for _changes API in sql-cloudant
> --
>
> Key: BAHIR-137
> URL: https://issues.apache.org/jira/browse/BAHIR-137
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Items for improving _changes feed load:
> - Make Spark streaming batch interval visible to the user for tuning based on 
> type/size of document and number of docs in database
> - Merge BAHIR-128: Improve stability of _changes receiver
> - Merge BAHIR-154: refactor sql-cloudant to use java-cloudant library



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-137) Load performance improvements for _changes API in sql-cloudant

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311318#comment-16311318
 ] 

ASF GitHub Bot commented on BAHIR-137:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/60#discussion_r159648190
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreConfigManager.scala
 ---
@@ -35,6 +35,7 @@ object JsonStoreConfigManager {
   private val CLOUDANT_PASSWORD_CONFIG = "cloudant.password"
   private val CLOUDANT_PROTOCOL_CONFIG = "cloudant.protocol"
   private val CLOUDANT_API_ENDPOINT = "cloudant.endpoint"
+  private val CLOUDANT_STREAMING_BATCH_INTERVAL = "cloudant.batchInterval"
   private val STORAGE_LEVEL_FOR_CHANGES_INDEX = "cloudant.storageLevel"
--- End diff --

This still says `cloudant.storageLevel` but the documentation was changed 
to `storageLevel`, shouldn't they be the same?
  


> Load performance improvements for _changes API in sql-cloudant
> --
>
> Key: BAHIR-137
> URL: https://issues.apache.org/jira/browse/BAHIR-137
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Items for improving _changes feed load:
> - Make Spark streaming batch interval visible to the user for tuning based on 
> type/size of document and number of docs in database
> - Merge BAHIR-128: Improve stability of _changes receiver
> - Merge BAHIR-154: refactor sql-cloudant to use java-cloudant library



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-137) Load performance improvements for _changes API in sql-cloudant

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311319#comment-16311319
 ] 

ASF GitHub Bot commented on BAHIR-137:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/60#discussion_r159649060
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/internal/ChangesReceiver.scala
 ---
@@ -72,6 +72,7 @@ class ChangesReceiver(config: CloudantChangesConfig)
 val status = headers.getOrElse("Status", IndexedSeq.empty)
 val errorMsg = "Error retrieving _changes feed " + 
config.getDbname + ": " + status(0)
 reportError(errorMsg, new CloudantException(errorMsg))
+stop(errorMsg)
--- End diff --

Does this call to `stop` result in the errorMsg being reported twice?


> Load performance improvements for _changes API in sql-cloudant
> --
>
> Key: BAHIR-137
> URL: https://issues.apache.org/jira/browse/BAHIR-137
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Items for improving _changes feed load:
> - Make Spark streaming batch interval visible to the user for tuning based on 
> type/size of document and number of docs in database
> - Merge BAHIR-128: Improve stability of _changes receiver
> - Merge BAHIR-154: refactor sql-cloudant to use java-cloudant library



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)