[jira] [Commented] (BAHIR-141) Support for Array[Byte] in SparkGCPCredentials.jsonServiceAccount
[ https://issues.apache.org/jira/browse/BAHIR-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323671#comment-16323671 ] ASF GitHub Bot commented on BAHIR-141: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/53 Refer to this link for build results (access rights to CI server needed): http://169.45.79.58:8080/job/bahir_spark_pr_builder/151/ > Support for Array[Byte] in SparkGCPCredentials.jsonServiceAccount > - > > Key: BAHIR-141 > URL: https://issues.apache.org/jira/browse/BAHIR-141 > Project: Bahir > Issue Type: Improvement > Components: Spark Streaming Connectors >Affects Versions: Spark-2.0.2, Spark-2.1.1, Spark-2.2.0 >Reporter: Ankit Agrahari >Priority: Minor > Fix For: Spark-2.0.2, Spark-2.2.0 > > > Existing implementation of SparkGCPCredentials.jsonServiceAccount has only > support for reading the credential file from given path in config(i.e local > path). If developer does not have access to worker node it makes easy for > developer to add the byte array of json service file rather than path(which > will not be available on worker node and will lead to FileNotFound > Exaception)from the machine where driver is submitting job from. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-141) Support for Array[Byte] in SparkGCPCredentials.jsonServiceAccount
[ https://issues.apache.org/jira/browse/BAHIR-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323670#comment-16323670 ] ASF GitHub Bot commented on BAHIR-141: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/53 :red_circle: Build failed, see build log for details > Support for Array[Byte] in SparkGCPCredentials.jsonServiceAccount > - > > Key: BAHIR-141 > URL: https://issues.apache.org/jira/browse/BAHIR-141 > Project: Bahir > Issue Type: Improvement > Components: Spark Streaming Connectors >Affects Versions: Spark-2.0.2, Spark-2.1.1, Spark-2.2.0 >Reporter: Ankit Agrahari >Priority: Minor > Fix For: Spark-2.0.2, Spark-2.2.0 > > > Existing implementation of SparkGCPCredentials.jsonServiceAccount has only > support for reading the credential file from given path in config(i.e local > path). If developer does not have access to worker node it makes easy for > developer to add the byte array of json service file rather than path(which > will not be available on worker node and will lead to FileNotFound > Exaception)from the machine where driver is submitting job from. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-138) Fix sql-cloudant deprecation messages
[
https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323662#comment-16323662
]
ASF GitHub Bot commented on BAHIR-138:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/63
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/150/
> Fix sql-cloudant deprecation messages
> -
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Minor
> Labels: warnings
> Fix For: Spark-2.2.1, Spark-2.3
>
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59:
> method json in class DataFrameReader is deprecated: Use json(Dataset[String])
> instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame =
> sqlContext.sparkSession.read.json(globalRDD)
> [WARNING]^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and
> {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to
> sql-cloudant/target/scala-2.11/test-classes...
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67:
> method registerTempTable in class Dataset is deprecated: Use
> createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING]^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-138) Fix sql-cloudant deprecation messages
[
https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323661#comment-16323661
]
ASF GitHub Bot commented on BAHIR-138:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/63
:red_circle: Build failed, see build log for details
> Fix sql-cloudant deprecation messages
> -
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Minor
> Labels: warnings
> Fix For: Spark-2.2.1, Spark-2.3
>
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59:
> method json in class DataFrameReader is deprecated: Use json(Dataset[String])
> instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame =
> sqlContext.sparkSession.read.json(globalRDD)
> [WARNING]^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and
> {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to
> sql-cloudant/target/scala-2.11/test-classes...
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67:
> method registerTempTable in class Dataset is deprecated: Use
> createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING]^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-138) Fix sql-cloudant deprecation messages
[
https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323652#comment-16323652
]
ASF GitHub Bot commented on BAHIR-138:
--
Github user lresende commented on the issue:
https://github.com/apache/bahir/pull/63
ok to test
> Fix sql-cloudant deprecation messages
> -
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Minor
> Labels: warnings
> Fix For: Spark-2.2.1, Spark-2.3
>
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59:
> method json in class DataFrameReader is deprecated: Use json(Dataset[String])
> instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame =
> sqlContext.sparkSession.read.json(globalRDD)
> [WARNING]^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and
> {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to
> sql-cloudant/target/scala-2.11/test-classes...
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67:
> method registerTempTable in class Dataset is deprecated: Use
> createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING]^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features
[
https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323564#comment-16323564
]
ASF GitHub Bot commented on BAHIR-154:
--
Github user emlaver commented on a diff in the pull request:
https://github.com/apache/bahir/pull/61#discussion_r161142828
--- Diff:
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/CloudantConfig.scala ---
@@ -16,34 +16,127 @@
*/
package org.apache.bahir.cloudant
-import java.net.URLEncoder
+import java.net.{URL, URLEncoder}
-import play.api.libs.json.{JsArray, JsObject, Json, JsValue}
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.reflect.io.File
+
+import com.cloudant.client.api.{ClientBuilder, CloudantClient, Database}
+import com.cloudant.client.api.model.SearchResult
+import com.cloudant.client.api.views._
+import com.cloudant.http.{Http, HttpConnection}
+import com.cloudant.http.interceptors.Replay429Interceptor
+import com.google.gson.{JsonObject, JsonParser}
import org.apache.bahir.cloudant.common._
+import org.apache.bahir.cloudant.common.JsonUtil.JsonConverter
/*
* Only allow one field pushdown now
* as the filter today does not tell how to link the filters out And v.s. Or
*/
class CloudantConfig(val protocol: String, val host: String,
- val dbName: String, val indexName: String, val
viewName: String)
+ val dbName: String, val indexPath: String, val
viewPath: String)
(implicit val username: String, val password: String,
val partitions: Int, val maxInPartition: Int, val
minInPartition: Int,
val requestTimeout: Long, val bulkSize: Int, val
schemaSampleSize: Int,
val createDBOnSave: Boolean, val endpoint: String,
val useQuery: Boolean = false, val queryLimit: Int)
extends Serializable {
+ @transient private lazy val client: CloudantClient = ClientBuilder
+.url(getClientUrl)
+.username(username)
+.password(password)
+.interceptors(Replay429Interceptor.WITH_DEFAULTS)
--- End diff --
Added option in 9fefcdc.
> Refactor sql-cloudant to use Cloudant's java-cloudant features
> --
>
> Key: BAHIR-154
> URL: https://issues.apache.org/jira/browse/BAHIR-154
> Project: Bahir
> Issue Type: Improvement
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>
> Cloudant's java-cloudant library (which is currently used for testing)
> contains several features that sql-cloudant can benefit from:
> - HTTP 429 backoff
> - View builder API to potentially simplify loading for _all_docs/views
> - Improved exception handling when executing HTTP requests
> - Future support for IAM API key
> Would need to replace current scala HTTP library with OkHttp library, and
> also replace play-json with GSON library.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-138) Fix sql-cloudant deprecation messages
[
https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323559#comment-16323559
]
ASF GitHub Bot commented on BAHIR-138:
--
GitHub user emlaver opened a pull request:
https://github.com/apache/bahir/pull/63
[BAHIR-138] fix deprecated warnings
_What_
Fix deprecated warnings in `DefaultSource` class.
_How_
- Imported `spark.implicits._` to convert Spark RDD to Dataset
- Replaced deprecated `json(RDD[String])` with `json(Dataset[String])`
See [BAHIR-138](https://issues.apache.org/jira/browse/BAHIR-138)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/emlaver/bahir 138-fix-deprecation-warnings
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bahir/pull/63.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #63
commit 458759e92e2015460e8994e30d98b72e1d661251
Author: Esteban Laver
Date: 2018-01-12T05:26:29Z
Fixed deprecation warning messages
- Imported ‘spark.implicits._’ to convert Spark RDD to Dataset
- Replaced deprecated `json(RDD[String])` with `json(Dataset[String])`
> Fix sql-cloudant deprecation messages
> -
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Minor
> Labels: warnings
> Fix For: Spark-2.2.1, Spark-2.3
>
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59:
> method json in class DataFrameReader is deprecated: Use json(Dataset[String])
> instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame =
> sqlContext.sparkSession.read.json(globalRDD)
> [WARNING]^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and
> {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to
> sql-cloudant/target/scala-2.11/test-classes...
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67:
> method registerTempTable in class Dataset is deprecated: Use
> createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING]^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-138) Fix sql-cloudant deprecation messages
[
https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323560#comment-16323560
]
ASF GitHub Bot commented on BAHIR-138:
--
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/63
Can one of the admins verify this patch?
> Fix sql-cloudant deprecation messages
> -
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
>Affects Versions: Spark-2.2.0
>Reporter: Esteban Laver
>Assignee: Esteban Laver
>Priority: Minor
> Labels: warnings
> Fix For: Spark-2.2.1, Spark-2.3
>
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59:
> method json in class DataFrameReader is deprecated: Use json(Dataset[String])
> instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame =
> sqlContext.sparkSession.read.json(globalRDD)
> [WARNING]^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and
> {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to
> sql-cloudant/target/scala-2.11/test-classes...
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67:
> method registerTempTable in class Dataset is deprecated: Use
> createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING]^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features
[ https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323495#comment-16323495 ] ASF GitHub Bot commented on BAHIR-154: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/61 Refer to this link for build results (access rights to CI server needed): http://169.45.79.58:8080/job/bahir_spark_pr_builder/149/ > Refactor sql-cloudant to use Cloudant's java-cloudant features > -- > > Key: BAHIR-154 > URL: https://issues.apache.org/jira/browse/BAHIR-154 > Project: Bahir > Issue Type: Improvement >Affects Versions: Spark-2.2.0 >Reporter: Esteban Laver >Assignee: Esteban Laver > > Cloudant's java-cloudant library (which is currently used for testing) > contains several features that sql-cloudant can benefit from: > - HTTP 429 backoff > - View builder API to potentially simplify loading for _all_docs/views > - Improved exception handling when executing HTTP requests > - Future support for IAM API key > Would need to replace current scala HTTP library with OkHttp library, and > also replace play-json with GSON library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features
[ https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323494#comment-16323494 ] ASF GitHub Bot commented on BAHIR-154: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/61 :white_check_mark: Build successful > Refactor sql-cloudant to use Cloudant's java-cloudant features > -- > > Key: BAHIR-154 > URL: https://issues.apache.org/jira/browse/BAHIR-154 > Project: Bahir > Issue Type: Improvement >Affects Versions: Spark-2.2.0 >Reporter: Esteban Laver >Assignee: Esteban Laver > > Cloudant's java-cloudant library (which is currently used for testing) > contains several features that sql-cloudant can benefit from: > - HTTP 429 backoff > - View builder API to potentially simplify loading for _all_docs/views > - Improved exception handling when executing HTTP requests > - Future support for IAM API key > Would need to replace current scala HTTP library with OkHttp library, and > also replace play-json with GSON library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Missing changes for BAHIR-138
Thanks Esteban, not sure what might have happened on the merge, but good catch, and I will merge those shortly. On Thu, Jan 11, 2018 at 11:24 AM, Esteban Laver wrote: > Hi all, > > I'll be opening up a new PR for BAHIR-138 as there were changes to the > DefaultSource class that existed in PR 59 (https://github.com/apache/ > bahir/pull/59) but somehow were not part of the commit when merged to > master: > > https://github.com/apache/bahir/commit/eae02f29eb011f50bc313714e6cde6 > 2ce65804c4 > > > Thanks, > Esteban (@emlaver) -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Missing changes for BAHIR-138
Hi all, I'll be opening up a new PR for BAHIR-138 as there were changes to the DefaultSource class that existed in PR 59 (https://github.com/apache/bahir/pull/59) but somehow were not part of the commit when merged to master: https://github.com/apache/bahir/commit/eae02f29eb011f50bc313714e6cde62ce65804c4 Thanks, Esteban (@emlaver)
[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features
[ https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322534#comment-16322534 ] ASF GitHub Bot commented on BAHIR-154: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/61 :white_check_mark: Build successful > Refactor sql-cloudant to use Cloudant's java-cloudant features > -- > > Key: BAHIR-154 > URL: https://issues.apache.org/jira/browse/BAHIR-154 > Project: Bahir > Issue Type: Improvement >Affects Versions: Spark-2.2.0 >Reporter: Esteban Laver >Assignee: Esteban Laver > > Cloudant's java-cloudant library (which is currently used for testing) > contains several features that sql-cloudant can benefit from: > - HTTP 429 backoff > - View builder API to potentially simplify loading for _all_docs/views > - Improved exception handling when executing HTTP requests > - Future support for IAM API key > Would need to replace current scala HTTP library with OkHttp library, and > also replace play-json with GSON library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-154) Refactor sql-cloudant to use Cloudant's java-cloudant features
[ https://issues.apache.org/jira/browse/BAHIR-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322535#comment-16322535 ] ASF GitHub Bot commented on BAHIR-154: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/61 Refer to this link for build results (access rights to CI server needed): http://169.45.79.58:8080/job/bahir_spark_pr_builder/148/ > Refactor sql-cloudant to use Cloudant's java-cloudant features > -- > > Key: BAHIR-154 > URL: https://issues.apache.org/jira/browse/BAHIR-154 > Project: Bahir > Issue Type: Improvement >Affects Versions: Spark-2.2.0 >Reporter: Esteban Laver >Assignee: Esteban Laver > > Cloudant's java-cloudant library (which is currently used for testing) > contains several features that sql-cloudant can benefit from: > - HTTP 429 backoff > - View builder API to potentially simplify loading for _all_docs/views > - Improved exception handling when executing HTTP requests > - Future support for IAM API key > Would need to replace current scala HTTP library with OkHttp library, and > also replace play-json with GSON library. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
