unsubscribe
Charles Bajomo
Operations Director
www.cloudxtiny.co.uk | Precision Technology Consulting Ltd
Registered England & Wales : 07397178
VAT No. : 124 4354 38 GB
The easiest way I found was to take a look at the source. Any receiver that
calls the version of store that requires an iterator is considered reliable. A
definitive list would be nice.
Kind Regards
- Original Message -
From: "Justin Pihony"
To: "user"
Sent: Monday, 17 April, 2017 20:
Hello all,
I have a strange behaviour I can't understand. I have a streaming job using a
custom java receiver that pull data from a jms queue that I process and then
write to HDFS as parquet and avro files. For some reason my job keeps failing
after 1hr and 30 minutes. When It fails I get an e
Hello everyone,
I wanted to know if there is any benefit to using an acculumator over just
executing a count() on the whole RDD. There seems to be a lot of issues with
accumulator during a stage failure and also seems to be an issue rebuilding
them if the application restarts from a checkpoint
Unless this is a managed hive table I would expect you can just MSCK REPAIR the
table to get the new partition. of course you will need to change the schema to
reflect the new partition
Kind Regards
From: "Triones,Deng(vip.com)"
To: "Charles O. Bajomo"
Cc: "user
I see this problem as well with the _temporary directory but from what I have
been able to gather, there is no way around it in that situation apart from
making sure all reducers write to different folders. In the past I partitioned
by executor id. I don't know if this is the best way though.
Hello,
I am reading data from a JMS queue and I need to prevent any data loss so I
have custom java receiver that only acks messages once they have been stored.
Sometimes my program crashes because I can't control the flow rate from the
queue and it overwhelms the job and I end up losing data.
Hello all,
I am having some problems with my custom java based receiver. I am running
Spark 1.5.0 and I used the template on the spark website
(http://spark.apache.org/docs/1.0.0/streaming-custom-receivers.html). Basically
my receiver listens to a JMS queue (Solace) and then based on the size