Hi,
You could also use this Receiver :
https://github.com/dibbhatt/kafka-spark-consumer
This is part of spark-packages also :
https://spark-packages.org/package/dibbhatt/kafka-spark-consumer
You do not need to enable WAL in this and still recover from Driver failure
with no data loss. You can re
IIUC, your scenario is quite like what currently ReliableKafkaReceiver
does. You can only send ack to the upstream source after WAL is persistent,
otherwise because of asynchronization of data processing and data
receiving, there's still a chance data could be lost if you send out ack
before WAL.
Hi All,
I am using a Receiver based approach. And I understand that spark streaming
API's will convert the received data from receiver into blocks and these
blocks that are in memory are also stored in WAL if one enables it. my
upstream source which is not Kafka can also replay by which I mean if
Is there any specific reason why this feature is only supported in
streaming?
On Wed, Jun 8, 2016 at 3:24 PM, Ted Yu wrote:
> There was a minor typo in the name of the config:
>
> spark.streaming.receiver.writeAheadLog.enable
>
> Yes, it only applies to Streaming.
>
> On Wed, Jun 8, 2016 at 3:14
There was a minor typo in the name of the config:
spark.streaming.receiver.writeAheadLog.enable
Yes, it only applies to Streaming.
On Wed, Jun 8, 2016 at 3:14 PM, Mohit Anchlia
wrote:
> Is something similar to park.streaming.receiver.writeAheadLog.enable
> available on SparkContext? It looks l
Is something similar to park.streaming.receiver.writeAheadLog.enable
available on SparkContext? It looks like it only works for spark streaming.
Thanks
> Saisai
>
>> On Tue, Mar 15, 2016 at 5:12 PM, Ewan Leith
>> wrote:
>> Has anyone seen a way of updating the Spark streaming job configuration
>> while retaining the existing data in the write ahead log?
>>
>>
>>
>> e.g. if you’ve
That’s what I thought, it’s a shame!
Thanks Saisai,
Ewan
From: Saisai Shao [mailto:sai.sai.s...@gmail.com]
Sent: 15 March 2016 09:22
To: Ewan Leith
Cc: user
Subject: Re: Spark streaming - update configuration while retaining write ahead
log data?
Currently configuration is a part of
x27;s no way to handle your situation.
Thanks
Saisai
On Tue, Mar 15, 2016 at 5:12 PM, Ewan Leith
wrote:
> Has anyone seen a way of updating the Spark streaming job configuration
> while retaining the existing data in the write ahead log?
>
>
>
> e.g. if you’ve launched a job w
Has anyone seen a way of updating the Spark streaming job configuration while
retaining the existing data in the write ahead log?
e.g. if you've launched a job without enough executors and a backlog has built
up in the WAL, can you increase the number of executors without losing the WAL
Hi spark user
I am running an spark streaming app that use receiver from a pubsub
system, and the pubsub system does NOT support ack.
And I don't want the data to be lost if there is a driver failure, and by
accident, the batches queue up at that time.
I tested by generating some queued ba
configure-checkpointing
>
> On Thu, Jan 21, 2016 at 3:32 AM, Patrick McGloin <
> mcgloin.patr...@gmail.com> wrote:
>
>> Hi all,
>>
>> To have a simple way of testing the Spark Streaming Write Ahead Log I
>> created a very simple Custom Input Receiver, which wi
le way of testing the Spark Streaming Write Ahead Log I
> created a very simple Custom Input Receiver, which will generate strings
> and store those:
>
> class InMemoryStringReceiver extends
> Receiver[String](StorageLevel.MEMORY_AND_DISK_SER) {
>
> val batchID = System.current
Hi all,
To have a simple way of testing the Spark Streaming Write Ahead Log I
created a very simple Custom Input Receiver, which will generate strings
and store those:
class InMemoryStringReceiver extends
Receiver[String](StorageLevel.MEMORY_AND_DISK_SER) {
val batchID
orks , does HDFS path gets bigger and bigger up
> everyday
> do I need to write an clean up job to delete data from write ahead logs
> folder ?
> what actually does write ahead log folder has ?
>
> Thanks
> Sri
>
>
>
>
> --
> View this message in context:
>
folder ?
what actually does write ahead log folder has ?
Thanks
Sri
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Does-spark-streaming-write-ahead-log-writes-all-received-data-to-HDFS-tp25439.html
Sent from the Apache Spark User List mailing list
16 matches
Mail list logo