[
https://issues.apache.org/jira/browse/BAHIR-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295212#comment-16295212
]
ASF GitHub Bot commented on BAHIR-128:
--------------------------------------
Github user emlaver commented on a diff in the pull request:
https://github.com/apache/bahir/pull/57#discussion_r157533814
--- Diff:
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/internal/ChangesReceiver.scala
---
@@ -39,56 +37,38 @@ class ChangesReceiver(config: CloudantChangesConfig)
}
private def receive(): Unit = {
- // Get total number of docs in database using _all_docs endpoint
- val limit = new JsonStoreDataAccess(config)
- .getTotalRows(config.getTotalUrl, queryUsed = false)
-
- // Get continuous _changes url
+ // Get normal _changes url
--- End diff --
For our internal implementation, we (myself and Mayya) wanted the user to
have a snapshot of data to load into Spark. For that to be possible, we
decided to use `continuous` style feed with a doc limit. With the new _changes
implementation from Mike's project, the `normal` feed is stable and works as
expected. I've also lowered the amount of requests/load time by removing the
HTTP request for the doc limit since it's not needed with `normal` style
_changes feed.
To work with data in "real-time", you can use `CloudantReciever` which
creates an eternal changes feed within the Spark Streaming context.
> Test failing sporadically in sql-cloudant's CloudantChangesDFSuite
> ------------------------------------------------------------------
>
> Key: BAHIR-128
> URL: https://issues.apache.org/jira/browse/BAHIR-128
> Project: Bahir
> Issue Type: Bug
> Reporter: Esteban Laver
> Assignee: Esteban Laver
> Priority: Minor
>
> This failure happened during pre-release testing for Bahir RC 2.2.0:
> CloudantChangesDFSuite:
> - load and save data from Cloudant database *** FAILED ***
> 0 did not equal 1967 (CloudantChangesDFSuite.scala:49)
> Partial stack trace:
> {code:java}
> Exception in thread "Cloudant Receiver" org.apache.spark.SparkException:
> Cannot add data as BlockGenerator has not been started or has been stopped
> at
> org.apache.spark.streaming.receiver.BlockGenerator.addData(BlockGenerator.scala:173)
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushSingle(ReceiverSupervisorImpl.scala:120)
> at org.apache.spark.streaming.receiver.Receiver.store(Receiver.scala:119)
> at
> org.apache.bahir.cloudant.internal.ChangesReceiver$$anonfun$org$apache$bahir$cloudant$internal$ChangesReceiver$$receive$1$$anonfun$apply$1.apply(ChangesReceiver.scala:82)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
