[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341180#comment-16341180
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user asfgit closed the pull request at:

https://github.com/apache/bahir/pull/61


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
> Fix For: Spark-2.2.1, Spark-2.3
>
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-26 Thread ASF subversion and git services (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341178#comment-16341178
 ] 

ASF subversion and git services commented on BAHIR-154:
---

Commit 6ea42a8965d98773a342ad5cd31aab6b64e9d9bd in bahir's branch 
refs/heads/master from [~emlaver]
[ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=6ea42a8 ]

[BAHIR-154] Refactor sql-cloudant to use cloudant-client library

- Use java-cloudant’s executeRequest for HTTP requests against
  _all_docs endpoint
- Added HTTP 429 backoff with default settings
- Simplified caught exception and message for schema size
- Replaced scala http library with okhttp library for changes receiver
- Updated streaming CloudantReceiver class to use improved
  ChangesRowScanner method
- Replaced Play JSON with GSON library
- Updated save operation to use java-cloudant bulk API
- Use _changes feed filter option for Cloudant/CouchDB 2.x and greater

Closes #61


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-24 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338850#comment-16338850
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on the issue:

https://github.com/apache/bahir/pull/61
  
@lresende PR rebased and fixed error message after rebasing #60.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-24 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338781#comment-16338781
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
:white_check_mark: Build successful
 



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-24 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338782#comment-16338782
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  

Refer to this link for build results (access rights to CI server needed): 
http://169.45.79.58:8080/job/bahir_spark_pr_builder/157/



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-24 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338753#comment-16338753
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
:red_circle: Build failed, see build log for details
 



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-24 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338754#comment-16338754
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  

Refer to this link for build results (access rights to CI server needed): 
http://169.45.79.58:8080/job/bahir_spark_pr_builder/156/



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-24 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338556#comment-16338556
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user lresende commented on the issue:

https://github.com/apache/bahir/pull/61
  
@emlaver Could you please rebase this pr


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Major
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-12 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324219#comment-16324219
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r161270944
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreDataAccess.scala
 ---
@@ -58,35 +54,30 @@ class JsonStoreDataAccess (config: CloudantConfig)  {
 }
   }
 
-  def getAll[T](url: String)
-  (implicit columns: Array[String] = null): Seq[String] = {
-this.getQueryResult[Seq[String]](url, processAll)
-  }
-
   def getIterator(skip: Int, limit: Int, url: String)
   (implicit columns: Array[String] = null,
   postData: String = null): Iterator[String] = {
+logger.info(s"Loading data from Cloudant using: $url , postData: 
$postData")
 val newUrl = config.getSubSetUrl(url, skip, limit, postData != null)
 this.getQueryResult[Iterator[String]](newUrl, processIterator)
   }
 
   def getTotalRows(url: String, queryUsed: Boolean)
-  (implicit postData: String = null): Int = {
-  if (queryUsed) config.queryLimit // Query can not retrieve total row 
now.
-  else {
-val totalUrl = config.getTotalUrl(url)
-this.getQueryResult[Int](totalUrl,
-  { result => config.getTotalRows(Json.parse(result))})
-  }
+  (implicit postData: String = null): Int = {
--- End diff --

is this a bit too indented?


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-11 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323564#comment-16323564
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r161142828
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
--- End diff --

Added option in 9fefcdc.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-11 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323495#comment-16323495
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  

Refer to this link for build results (access rights to CI server needed): 
http://169.45.79.58:8080/job/bahir_spark_pr_builder/149/



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-11 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323494#comment-16323494
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
:white_check_mark: Build successful
 



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-11 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322534#comment-16322534
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
:white_check_mark: Build successful
 



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-11 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322535#comment-16322535
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  

Refer to this link for build results (access rights to CI server needed): 
http://169.45.79.58:8080/job/bahir_spark_pr_builder/148/



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-09 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318768#comment-16318768
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160470451
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
--- End diff --

Also the defaults may be insufficient for some use cases. It may be worth 
exposing config option(s) for at least the number of retries if not the backoff.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315700#comment-16315700
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  

Refer to this link for build results (access rights to CI server needed): 
http://169.45.79.58:8080/job/bahir_spark_pr_builder/147/



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315699#comment-16315699
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
:white_check_mark: Build successful
 



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315693#comment-16315693
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077466
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
--- End diff --

Improved regex pattern in in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315694#comment-16315694
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077474
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
--- End diff --

Improved regex pattern in in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315688#comment-16315688
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077413
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
+}
+  }
+
+  def getDocs(limit: Int): List[JsonObject] = {
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
--- End diff --

Removed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315692#comment-16315692
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077433
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
+}
+  }
+
+  def getDocs(limit: Int): List[JsonObject] = {
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  
buildViewRequest(limit).build().getResponse.getDocsAs(classOf[JsonObject]).asScala.toList
+} else if (indexPath != null) {
+  var searchDocs = mutable.ListBuffer[JsonObject]()
+  for (result: SearchResult[JsonObject]#SearchResultRow <-
+   buildSearchRequest(limit).getRows.asScala) {
+searchDocs += result.getDoc
+  }
+  searchDocs.toList
+} else {
+  // /_all_docs?limit=1
+  // val response = client.executeRequest(Http.GET(
--- End diff --

Removed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315687#comment-16315687
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077412
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -101,24 +105,24 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
   case LessThanOrEqual(attr, v) => ("$lte", v)
   case _ => (null, null)
 }
-val convertedV: JsValue = {
+val convertedV: JsonElement = {
   // TODO Better handing of other types
   if (value != null) {
 value match {
-  case s: String => Json.toJson(s)
-  case l: Long => Json.toJson(l)
-  case d: Double => Json.toJson(d)
-  case i: Int => Json.toJson(i)
-  case b: Boolean => Json.toJson(b)
-  case t: java.sql.Timestamp => Json.toJson(t)
+  case s: String => parser.parse(s)// Json.toJson(s)
--- End diff --

Removed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315690#comment-16315690
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077423
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
+  // return 'design-doc/search-name'
+  splitPath(1) + File.separator + splitPath(3)
+} else {
+  null
+}
+  }
+  lazy val viewName: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  val splitViewPath = viewPath.split(File.separator)
+  if(splitViewPath(3).contains("?")) {
+splitViewPath(3).substring(0, splitViewPath(3).indexOf("?"))
+  } else {
+splitViewPath(3)
+  }
+} else {
+  null
+}
+  }
 
   val pkField = "_id"
   val defaultIndex: String = endpoint
   val default_filter: String = "*:*"
 
-  def getDbUrl: String = {
-dbUrl
+  def buildAllDocsRequest(limit: Int, includeDocs: Boolean = true): 
AllDocsRequestBuilder = {
+var allDocsReq = 
database.getAllDocsRequestBuilder.includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  allDocsReq = allDocsReq.limit(limit)
+}
+allDocsReq
+  }
+
+  def buildViewRequest(limit: Int, includeDocs: Boolean = true):
+  UnpaginatedRequestBuilder[String, String] = {
+val viewReq = database.getViewRequestBuilder(designDoc, viewName)
+  .newRequest(Key.Type.STRING, classOf[String])
+  .includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  viewReq.limit(limit)
+}
+viewReq
+  }
+
+  def buildSearchRequest(limit: Int): SearchResult[JsonObject] = {
+val searchReq = database.search(searchName)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  searchReq.limit(limit)
+}
+searchReq.querySearchResult(default_filter, classOf[JsonObject])
+  }
+
+  def executeRequest(stringUrl: String, postData: String = null): 
HttpConnection = {
+val url = new URL(stringUrl)
+if(postData != 

[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315686#comment-16315686
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077409
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -174,7 +178,11 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
 
 implicit val postData : String = {
   if (queryUsed) {
-Json.stringify(Json.obj("selector" -> selector, "limit" -> 1))
+val jsonSelector = new JsonObject
+jsonSelector.addProperty("selector", selector.toString)
+jsonSelector.addProperty("limit", 1)
+jsonSelector.toString
+// Json.stringify(Json.obj("selector" -> selector, "limit" -> 1))
--- End diff --

Removed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315684#comment-16315684
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077401
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -191,20 +199,30 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
(0 until totalPartition).map(i => {
   val skip = i * limitPerPartition
   new JsonStoreRDDPartition(url, skip, limitPerPartition, i,
-  config, selector, fields, queryUsed).asInstanceOf[Partition]
+  config, selector, fields, queryUsed)
+.asInstanceOf[Partition]
 }).toArray
   }
 
   override def compute(splitIn: Partition, context: TaskContext):
   Iterator[String] = {
 val myPartition = splitIn.asInstanceOf[JsonStoreRDDPartition]
 implicit val postData : String = {
+  val jsonObject = new JsonObject
   if (myPartition.queryUsed && myPartition.fields != null) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" -> myPartition.fields,
-"limit" -> myPartition.limit, "skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" ->
+// myPartition.fields, "limit" -> myPartition.limit, "skip" -> 
myPartition.skip))
+jsonObject.add("selector", myPartition.selector)
+jsonObject.add("fields", myPartition.fields)
+jsonObject.addProperty("skip", myPartition.skip)
+jsonObject.toString
   } else if (myPartition.queryUsed) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"limit" -> myPartition.limit,
-"skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector,
+// "limit" -> myPartition.limit, "skip" -> myPartition.skip))
--- End diff --

Removed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315685#comment-16315685
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077403
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -191,20 +199,30 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
(0 until totalPartition).map(i => {
   val skip = i * limitPerPartition
   new JsonStoreRDDPartition(url, skip, limitPerPartition, i,
-  config, selector, fields, queryUsed).asInstanceOf[Partition]
+  config, selector, fields, queryUsed)
+.asInstanceOf[Partition]
 }).toArray
   }
 
   override def compute(splitIn: Partition, context: TaskContext):
   Iterator[String] = {
 val myPartition = splitIn.asInstanceOf[JsonStoreRDDPartition]
 implicit val postData : String = {
+  val jsonObject = new JsonObject
   if (myPartition.queryUsed && myPartition.fields != null) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" -> myPartition.fields,
-"limit" -> myPartition.limit, "skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" ->
+// myPartition.fields, "limit" -> myPartition.limit, "skip" -> 
myPartition.skip))
--- End diff --

Removed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-07 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315683#comment-16315683
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r160077380
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -191,20 +199,30 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
(0 until totalPartition).map(i => {
   val skip = i * limitPerPartition
   new JsonStoreRDDPartition(url, skip, limitPerPartition, i,
-  config, selector, fields, queryUsed).asInstanceOf[Partition]
+  config, selector, fields, queryUsed)
+.asInstanceOf[Partition]
 }).toArray
   }
 
   override def compute(splitIn: Partition, context: TaskContext):
   Iterator[String] = {
 val myPartition = splitIn.asInstanceOf[JsonStoreRDDPartition]
 implicit val postData : String = {
+  val jsonObject = new JsonObject
   if (myPartition.queryUsed && myPartition.fields != null) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" -> myPartition.fields,
-"limit" -> myPartition.limit, "skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" ->
+// myPartition.fields, "limit" -> myPartition.limit, "skip" -> 
myPartition.skip))
+jsonObject.add("selector", myPartition.selector)
+jsonObject.add("fields", myPartition.fields)
+jsonObject.addProperty("skip", myPartition.skip)
+jsonObject.toString
   } else if (myPartition.queryUsed) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"limit" -> myPartition.limit,
-"skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector,
+// "limit" -> myPartition.limit, "skip" -> myPartition.skip))
+jsonObject.add("selector", myPartition.selector)
--- End diff --

Fixed in 55cc844.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313229#comment-16313229
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159845718
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
+  // return 'design-doc/search-name'
+  splitPath(1) + File.separator + splitPath(3)
+} else {
+  null
+}
+  }
+  lazy val viewName: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  val splitViewPath = viewPath.split(File.separator)
+  if(splitViewPath(3).contains("?")) {
+splitViewPath(3).substring(0, splitViewPath(3).indexOf("?"))
+  } else {
+splitViewPath(3)
+  }
+} else {
+  null
+}
+  }
 
   val pkField = "_id"
   val defaultIndex: String = endpoint
   val default_filter: String = "*:*"
 
-  def getDbUrl: String = {
-dbUrl
+  def buildAllDocsRequest(limit: Int, includeDocs: Boolean = true): 
AllDocsRequestBuilder = {
+var allDocsReq = 
database.getAllDocsRequestBuilder.includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  allDocsReq = allDocsReq.limit(limit)
+}
+allDocsReq
+  }
+
+  def buildViewRequest(limit: Int, includeDocs: Boolean = true):
+  UnpaginatedRequestBuilder[String, String] = {
+val viewReq = database.getViewRequestBuilder(designDoc, viewName)
+  .newRequest(Key.Type.STRING, classOf[String])
+  .includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  viewReq.limit(limit)
+}
+viewReq
+  }
+
+  def buildSearchRequest(limit: Int): SearchResult[JsonObject] = {
+val searchReq = database.search(searchName)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  searchReq.limit(limit)
+}
+searchReq.querySearchResult(default_filter, classOf[JsonObject])
+  }
+
+  def executeRequest(stringUrl: String, postData: String = null): 
HttpConnection = {
+val url = new URL(stringUrl)
+if(postData != 

[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313237#comment-16313237
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159889840
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -191,20 +199,30 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
(0 until totalPartition).map(i => {
   val skip = i * limitPerPartition
   new JsonStoreRDDPartition(url, skip, limitPerPartition, i,
-  config, selector, fields, queryUsed).asInstanceOf[Partition]
+  config, selector, fields, queryUsed)
+.asInstanceOf[Partition]
 }).toArray
   }
 
   override def compute(splitIn: Partition, context: TaskContext):
   Iterator[String] = {
 val myPartition = splitIn.asInstanceOf[JsonStoreRDDPartition]
 implicit val postData : String = {
+  val jsonObject = new JsonObject
   if (myPartition.queryUsed && myPartition.fields != null) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" -> myPartition.fields,
-"limit" -> myPartition.limit, "skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" ->
+// myPartition.fields, "limit" -> myPartition.limit, "skip" -> 
myPartition.skip))
+jsonObject.add("selector", myPartition.selector)
+jsonObject.add("fields", myPartition.fields)
+jsonObject.addProperty("skip", myPartition.skip)
+jsonObject.toString
   } else if (myPartition.queryUsed) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"limit" -> myPartition.limit,
-"skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector,
+// "limit" -> myPartition.limit, "skip" -> myPartition.skip))
--- End diff --

remove


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313238#comment-16313238
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159890761
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -191,20 +199,30 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
(0 until totalPartition).map(i => {
   val skip = i * limitPerPartition
   new JsonStoreRDDPartition(url, skip, limitPerPartition, i,
-  config, selector, fields, queryUsed).asInstanceOf[Partition]
+  config, selector, fields, queryUsed)
+.asInstanceOf[Partition]
 }).toArray
   }
 
   override def compute(splitIn: Partition, context: TaskContext):
   Iterator[String] = {
 val myPartition = splitIn.asInstanceOf[JsonStoreRDDPartition]
 implicit val postData : String = {
+  val jsonObject = new JsonObject
   if (myPartition.queryUsed && myPartition.fields != null) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" -> myPartition.fields,
-"limit" -> myPartition.limit, "skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" ->
+// myPartition.fields, "limit" -> myPartition.limit, "skip" -> 
myPartition.skip))
+jsonObject.add("selector", myPartition.selector)
+jsonObject.add("fields", myPartition.fields)
+jsonObject.addProperty("skip", myPartition.skip)
+jsonObject.toString
   } else if (myPartition.queryUsed) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"limit" -> myPartition.limit,
-"skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector,
+// "limit" -> myPartition.limit, "skip" -> myPartition.skip))
+jsonObject.add("selector", myPartition.selector)
--- End diff --

Could add the `selector` and `skip` before the `if`/`else` and avoid the 
duplication


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313233#comment-16313233
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159847773
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -54,20 +147,21 @@ class CloudantConfig(val protocol: String, val host: 
String,
 createDBOnSave
   }
 
-  def getLastNum(result: JsValue): JsValue = (result \ "last_seq").get
+  def getClientUrl: URL = {
+new URL(protocol + "://" + host)
+  }
+
+  def getLastNum(result: JsonObject): JsonObject = 
result.get("last_seq").getAsJsonObject
 
   /* Url containing limit for docs in a Cloudant database.
   * If a view is not defined, use the _all_docs endpoint.
   * @return url with one doc limit for retrieving total doc count
   */
   def getUrl(limit: Int, excludeDDoc: Boolean = false): String = {
--- End diff --

The comment suggests that this method is used to get a URL with a `limit=1` 
for the purposes of getting a doc count, but then it is not used by 
`getTotalDocCount` it does however appear to be used by `getMany` with a 
different limit for query results.
FWIW I can't see `getTotalDocCount` being called anywhere in the new code 
either.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313239#comment-16313239
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159889787
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -191,20 +199,30 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
(0 until totalPartition).map(i => {
   val skip = i * limitPerPartition
   new JsonStoreRDDPartition(url, skip, limitPerPartition, i,
-  config, selector, fields, queryUsed).asInstanceOf[Partition]
+  config, selector, fields, queryUsed)
+.asInstanceOf[Partition]
 }).toArray
   }
 
   override def compute(splitIn: Partition, context: TaskContext):
   Iterator[String] = {
 val myPartition = splitIn.asInstanceOf[JsonStoreRDDPartition]
 implicit val postData : String = {
+  val jsonObject = new JsonObject
   if (myPartition.queryUsed && myPartition.fields != null) {
-Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" -> myPartition.fields,
-"limit" -> myPartition.limit, "skip" -> myPartition.skip))
+// Json.stringify(Json.obj("selector" -> myPartition.selector, 
"fields" ->
+// myPartition.fields, "limit" -> myPartition.limit, "skip" -> 
myPartition.skip))
--- End diff --

remove


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313230#comment-16313230
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159845769
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
+  // return 'design-doc/search-name'
+  splitPath(1) + File.separator + splitPath(3)
+} else {
+  null
+}
+  }
+  lazy val viewName: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  val splitViewPath = viewPath.split(File.separator)
+  if(splitViewPath(3).contains("?")) {
+splitViewPath(3).substring(0, splitViewPath(3).indexOf("?"))
+  } else {
+splitViewPath(3)
+  }
+} else {
+  null
+}
+  }
 
   val pkField = "_id"
   val defaultIndex: String = endpoint
   val default_filter: String = "*:*"
 
-  def getDbUrl: String = {
-dbUrl
+  def buildAllDocsRequest(limit: Int, includeDocs: Boolean = true): 
AllDocsRequestBuilder = {
+var allDocsReq = 
database.getAllDocsRequestBuilder.includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  allDocsReq = allDocsReq.limit(limit)
+}
+allDocsReq
+  }
+
+  def buildViewRequest(limit: Int, includeDocs: Boolean = true):
+  UnpaginatedRequestBuilder[String, String] = {
+val viewReq = database.getViewRequestBuilder(designDoc, viewName)
+  .newRequest(Key.Type.STRING, classOf[String])
+  .includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  viewReq.limit(limit)
+}
+viewReq
+  }
+
+  def buildSearchRequest(limit: Int): SearchResult[JsonObject] = {
+val searchReq = database.search(searchName)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  searchReq.limit(limit)
+}
+searchReq.querySearchResult(default_filter, classOf[JsonObject])
+  }
+
+  def executeRequest(stringUrl: String, postData: String = null): 
HttpConnection = {
+val url = new URL(stringUrl)
+if(postData != 

[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313236#comment-16313236
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159845855
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
+  // return 'design-doc/search-name'
+  splitPath(1) + File.separator + splitPath(3)
+} else {
+  null
+}
+  }
+  lazy val viewName: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  val splitViewPath = viewPath.split(File.separator)
+  if(splitViewPath(3).contains("?")) {
+splitViewPath(3).substring(0, splitViewPath(3).indexOf("?"))
+  } else {
+splitViewPath(3)
+  }
+} else {
+  null
+}
+  }
 
   val pkField = "_id"
   val defaultIndex: String = endpoint
   val default_filter: String = "*:*"
 
-  def getDbUrl: String = {
-dbUrl
+  def buildAllDocsRequest(limit: Int, includeDocs: Boolean = true): 
AllDocsRequestBuilder = {
+var allDocsReq = 
database.getAllDocsRequestBuilder.includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  allDocsReq = allDocsReq.limit(limit)
+}
+allDocsReq
+  }
+
+  def buildViewRequest(limit: Int, includeDocs: Boolean = true):
+  UnpaginatedRequestBuilder[String, String] = {
+val viewReq = database.getViewRequestBuilder(designDoc, viewName)
+  .newRequest(Key.Type.STRING, classOf[String])
+  .includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  viewReq.limit(limit)
+}
+viewReq
+  }
+
+  def buildSearchRequest(limit: Int): SearchResult[JsonObject] = {
+val searchReq = database.search(searchName)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  searchReq.limit(limit)
+}
+searchReq.querySearchResult(default_filter, classOf[JsonObject])
+  }
+
+  def executeRequest(stringUrl: String, postData: String = null): 
HttpConnection = {
+val url = new URL(stringUrl)
+if(postData != 

[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313228#comment-16313228
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159845032
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
--- End diff --

It may be worth adding an additional interceptor here to change/augment the 
UA.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313231#comment-16313231
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159848274
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
+}
+  }
+
+  def getDocs(limit: Int): List[JsonObject] = {
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
--- End diff --

remove?


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313232#comment-16313232
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159845874
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
+  // return 'design-doc/search-name'
+  splitPath(1) + File.separator + splitPath(3)
+} else {
+  null
+}
+  }
+  lazy val viewName: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  val splitViewPath = viewPath.split(File.separator)
+  if(splitViewPath(3).contains("?")) {
+splitViewPath(3).substring(0, splitViewPath(3).indexOf("?"))
+  } else {
+splitViewPath(3)
+  }
+} else {
+  null
+}
+  }
 
   val pkField = "_id"
   val defaultIndex: String = endpoint
   val default_filter: String = "*:*"
 
-  def getDbUrl: String = {
-dbUrl
+  def buildAllDocsRequest(limit: Int, includeDocs: Boolean = true): 
AllDocsRequestBuilder = {
+var allDocsReq = 
database.getAllDocsRequestBuilder.includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  allDocsReq = allDocsReq.limit(limit)
+}
+allDocsReq
+  }
+
+  def buildViewRequest(limit: Int, includeDocs: Boolean = true):
+  UnpaginatedRequestBuilder[String, String] = {
+val viewReq = database.getViewRequestBuilder(designDoc, viewName)
+  .newRequest(Key.Type.STRING, classOf[String])
+  .includeDocs(includeDocs)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  viewReq.limit(limit)
+}
+viewReq
+  }
+
+  def buildSearchRequest(limit: Int): SearchResult[JsonObject] = {
+val searchReq = database.search(searchName)
+if (limit != JsonStoreConfigManager.ALLDOCS_OR_CHANGES_LIMIT) {
+  searchReq.limit(limit)
+}
+searchReq.querySearchResult(default_filter, classOf[JsonObject])
+  }
+
+  def executeRequest(stringUrl: String, postData: String = null): 
HttpConnection = {
+val url = new URL(stringUrl)
+if(postData != 

[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313234#comment-16313234
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159889644
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -174,7 +178,11 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
 
 implicit val postData : String = {
   if (queryUsed) {
-Json.stringify(Json.obj("selector" -> selector, "limit" -> 1))
+val jsonSelector = new JsonObject
+jsonSelector.addProperty("selector", selector.toString)
+jsonSelector.addProperty("limit", 1)
+jsonSelector.toString
+// Json.stringify(Json.obj("selector" -> selector, "limit" -> 1))
--- End diff --

remove


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313235#comment-16313235
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159889542
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/common/JsonStoreRDD.scala 
---
@@ -101,24 +105,24 @@ class JsonStoreRDD(sc: SparkContext, config: 
CloudantConfig)
   case LessThanOrEqual(attr, v) => ("$lte", v)
   case _ => (null, null)
 }
-val convertedV: JsValue = {
+val convertedV: JsonElement = {
   // TODO Better handing of other types
   if (value != null) {
 value match {
-  case s: String => Json.toJson(s)
-  case l: Long => Json.toJson(l)
-  case d: Double => Json.toJson(d)
-  case i: Int => Json.toJson(i)
-  case b: Boolean => Json.toJson(b)
-  case t: java.sql.Timestamp => Json.toJson(t)
+  case s: String => parser.parse(s)// Json.toJson(s)
--- End diff --

some old code here that should probably be removed (and on the following 
line)


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312523#comment-16312523
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user emlaver commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159814009
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
--- End diff --

This is a great idea and I'll use this for both view and search.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311722#comment-16311722
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159699042
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
--- End diff --

I think you could leverage scala's raw interpreter here to make the regex a 
bit less escape-y:
`raw"\w+\/\w+\/\w+\/\w+"`

Also are word characters sufficient here? Aren't there some other 
characters that could be part of a design document id and allowed unencoded in 
a URL path (e.g. maybe `@`?) that would be excluded by this regex? I'd be 
inclined to maybe use something like: `_design\/(^\/)+\/_search\/(^\/)+`


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311723#comment-16311723
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159699798
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
  */
 package org.apache.bahir.cloudant
 
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
 
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
 
 import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
 
 /*
 * Only allow one field pushdown now
 * as the filter today does not tell how to link the filters out And v.s. Or
 */
 
 class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val 
viewName: String)
+ val dbName: String, val indexPath: String, val 
viewPath: String)
 (implicit val username: String, val password: String,
  val partitions: Int, val maxInPartition: Int, val 
minInPartition: Int,
  val requestTimeout: Long, val bulkSize: Int, val 
schemaSampleSize: Int,
  val createDBOnSave: Boolean, val endpoint: String,
  val useQuery: Boolean = false, val queryLimit: Int)
   extends Serializable {
 
+  @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
+.build
+  @transient private lazy val database: Database = client.database(dbName, 
false)
   lazy val dbUrl: String = {protocol + "://" + host + "/" + dbName}
+  lazy val designDoc: String = {
+if (viewPath != null && viewPath.nonEmpty) {
+  viewPath.split("/")(1)
+} else {
+null
+}
+  }
+  lazy val searchName: String = {
+// verify that the index path matches '_design/ddoc/_search/searchname'
+if (indexPath != null && indexPath.nonEmpty && 
indexPath.matches("\\w+\\/\\w+\\/\\w+\\/\\w+")) {
+  val splitPath = indexPath.split(File.separator)
--- End diff --

If you separate out the regex pattern from earlier you could use the 
captured groups to extract the design doc ID and search index name without 
needing to do more splits here.


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311721#comment-16311721
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159709935
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
+}
+  }
+
+  def getDocs(limit: Int): List[JsonObject] = {
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  
buildViewRequest(limit).build().getResponse.getDocsAs(classOf[JsonObject]).asScala.toList
+} else if (indexPath != null) {
+  var searchDocs = mutable.ListBuffer[JsonObject]()
+  for (result: SearchResult[JsonObject]#SearchResultRow <-
+   buildSearchRequest(limit).getRows.asScala) {
+searchDocs += result.getDoc
+  }
+  searchDocs.toList
+} else {
+  // /_all_docs?limit=1
+  // val response = client.executeRequest(Http.GET(
--- End diff --

Remove commented out code?


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-04 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311720#comment-16311720
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ricellis commented on a diff in the pull request:

https://github.com/apache/bahir/pull/61#discussion_r159709382
  
--- Diff: 
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -95,15 +189,49 @@ class CloudantConfig(val protocol: String, val host: 
String,
 }
   }
 
+  def getTotalDocCount: Int = {
+val limit = 1
+if (viewPath != null) {
+  // "limit=" + limit + "=" + skip
+  buildViewRequest(limit, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+} else {
+  // /_all_docs?limit=1
+  // Note: java-cloudant's AllDocsRequest doesn't have a 
getTotalRowCount method
+  // buildAllDocsRequest(1, includeDocs = 
false).build().getResponse.getTotalRowCount.toInt
+  val response = client.executeRequest(Http.GET(
+new URL(database.getDBUri + File.separator + endpoint + "?limit=" 
+ limit)))
+  getResultTotalRows(response.responseAsString)
--- End diff --

It might be easier to use
`com.cloudant.client.api.model.DbInfo#getDocCount` instead of trying to do 
it via `_all_docs`


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-03 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310605#comment-16310605
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
:white_check_mark: Build successful
 



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-03 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310606#comment-16310606
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  

Refer to this link for build results (access rights to CI server needed): 
http://169.45.79.58:8080/job/bahir_spark_pr_builder/145/



> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-03 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310599#comment-16310599
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user lresende commented on the issue:

https://github.com/apache/bahir/pull/61
  
ok to test


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-03 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310218#comment-16310218
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

Github user ApacheBahir commented on the issue:

https://github.com/apache/bahir/pull/61
  
Can one of the admins verify this patch?


> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features

2018-01-03 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310217#comment-16310217
 ] 

ASF GitHub Bot commented on BAHIR-154:
--

GitHub user emlaver opened a pull request:

https://github.com/apache/bahir/pull/61

[BAHIR-154] Refactor sql-cloudant to use Cloudant's java-cloudant features

_What_ 
Refactor sql-cloudant to use Cloudant's` java-cloudant` features

_How_
- Use java-cloudant’s executeRequest for HTTP requests against _all_docs 
endpoint
- Added HTTP 429 backoff with default settings
- Simplified caught exception and message for schema size
- Replaced scala http library with okhttp library for changes receiver
- Updated streaming CloudantReceiver class to use improved 
ChangesRowScanner method
- Replaced Play JSON with GSON library
- Updated save operation to use `java-cloudant` bulk API
- Use `_changes` feed filter option for Cloudant/CouchDB 2.x and greater

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/emlaver/bahir 154-java-cloudant-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bahir/pull/61.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #61


commit 72bdc4d960fc85bf22db8fbd655b12346118998e
Author: Esteban Laver 
Date:   2017-12-17T18:30:04Z

Initial commit for java-cloudant rebase
- Use java-cloudant’s executeRequest for HTTP requests against _all_docs 
endpoint
- Added HTTP 429 backoff with default settings
- Use java-cloudant view builders to get total doc limit
- Simplified caught exception and message for schema size

commit eb4053fe013dec1f3538224dee692118200ed19c
Author: Esteban Laver 
Date:   2017-12-09T04:55:01Z

Replaced scala http library with okhttp library for changes receiver

commit 5f20a3fd8d7e26a0c8c6ef9200d4386eeec31036
Author: Esteban Laver 
Date:   2017-12-17T17:54:47Z

Updated streaming CloudantReceiver class to use improved ChangesRowScanner 
method

commit 1dc43cc89bedfb241c41acd5961cf13d741fab86
Author: Esteban Laver 
Date:   2017-12-17T18:37:14Z

Replaced all references of play json with GSON library
- Updated save operation to use java-cloudant’s bulk API

commit 6520385e45634d23650c72a0431fe14d55762104
Author: Esteban Laver 
Date:   2017-12-19T17:02:26Z

Only use _changes feed filter option for Cloudant/CouchDB 2.x and greater




> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
>  Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing) 
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and 
> also replace play-json with GSON library.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)