[
https://issues.apache.org/jira/browse/BLUR-387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181021#comment-14181021
]
Dibyendu Bhattacharya edited comment on BLUR-387 at 10/23/14 5:07 AM:
----------------------------------------------------------------------
This Spark Blur connector index Kafka Messages to Apache Blur using following
Kafka Consumer.
https://github.com/dibbhatt/kafka-spark-consumer
This Fault Tolerant Kafka Consumer uses Low Level Kafka API to pull messages
from Kafka Topic Partition using Spark Custom Receiver.
Spark Blur Connector use this Kafka Consumer to index Kafka Messages using
Spark Hadoop APIs.
The Kafka DStream is repartitioned into number of partitions which is same as
number of Shards of Target Blur Table.
This connector uses a Custom Spark Partitioner to map keys to correct RDD
partition which intern maps to same Blur Shard.
This code is also available here :
https://github.com/dibbhatt/spark-blur-connector
was (Author: dibbhatt):
This Spark Blur connector index Kafka Messages to Apache Blur using following
Kafka Consumer.
https://github.com/dibbhatt/kafka-spark-consumer
This Fault Tolerant Kafka Consumer uses Low Level Kafka API to pull messages
from Kafka Topic Partition using Spark Custom Receiver.
For more details please refer to :
https://github.com/dibbhatt/kafka-spark-consumer
Spark Blur Connector use this Kafka Consumer to index Kafka Messages using
Spark Hadoop APIs.
The Kafka DStream is repartitioned into number of partitions which is same as
number of Shards of Target Blur Table.
This connector uses a Custom Spark Partitioner to map keys to correct RDD
partition which intern maps to same Blur Shard.
> Blur Spark Connector
> --------------------
>
> Key: BLUR-387
> URL: https://issues.apache.org/jira/browse/BLUR-387
> Project: Apache Blur
> Issue Type: New Feature
> Reporter: Dibyendu Bhattacharya
> Attachments: spark-blur-connector.rar
>
>
> Integrate Apache BLUR with Spark Streaming / Spark .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
