Hi,
How is the ContextCleaner different from spark.cleaner.ttl?Is
spark.cleaner.ttl when there is ContextCleaner in the Streaming job?
Thanks,
Swetha
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/broadcast-variable-get-cleaned-by-ContextCleaner
Hi, all
When I run some Spark application (actually unit test of the application in
Jenkins ), I found that I always hit the FileNotFoundException when reading
broadcast variable
The program itself works well, except the unit test
Here is the example log:
14/07/21 19:49:13 INFO
Hi, TD,
Thanks for the reply
I tried to reproduce this in a simpler program, but no luck
However, the program has been very simple, just load some files from HDFS and
write them to HBase….
---
It seems that the issue only appears when I run the unit test in Jenkins (not
fail every time,
Hi, TD,
I think I got more insights to the problem
in the Jenkins test file, I mistakenly pass a wrong value to spark.cores.max,
which is much larger than the expected value
(I passed master address as local[6], and spark.core.max as 200)
If I set a more consistent value, everything goes
That is definitely weird. spark.core.max should not affect thing when they
are running local mode.
And, I am trying to think of scenarios that could cause a broadcast
variable used in the current job to fall out of scope, but they all seem
very far fetched. So i am really curious to see the code
Ah, sorry, sorry, my brain just damaged….. sent some wrong information
not “spark.cores.max” but the minPartitions in sc.textFile()
Best,
--
Nan Zhu
On Monday, July 21, 2014 at 7:17 PM, Tathagata Das wrote:
That is definitely weird. spark.core.max should not affect thing when they
