The streaming job has been running ok in 1.2 and 1.3. After I upgraded to
1.4, I started seeing error as below. It appears that it fails in validate
method in StreamingContext. Is there anything changed on 1.4.0 w.r.t
DStream checkpointint?
Detailed error from driver:
15/07/15 18:00:39 ERROR
Can you show us your function(s) ?
Thanks
On Wed, Jul 15, 2015 at 12:46 PM, Chen Song chen.song...@gmail.com wrote:
The streaming job has been running ok in 1.2 and 1.3. After I upgraded to
1.4, I started seeing error as below. It appears that it fails in validate
method in StreamingContext.
Your streaming job may have been seemingly running ok, but the DStream
checkpointing must have been failing in the background. It would have been
visible in the log4j logs. In 1.4.0, we enabled fast-failure for that so
that checkpointing failures dont get hidden in the background.
The fact that
Ah, cool. Thanks.
On Wed, Jul 15, 2015 at 5:58 PM, Tathagata Das t...@databricks.com wrote:
Spark 1.4.1 just got released! So just download that. Yay for timing.
On Wed, Jul 15, 2015 at 2:47 PM, Ted Yu yuzhih...@gmail.com wrote:
Should be this one:
[SPARK-7180] [SPARK-8090]
Thanks
Can you point me to the patch to fix the serialization stack? Maybe I can
pull it in and rerun my job.
Chen
On Wed, Jul 15, 2015 at 4:40 PM, Tathagata Das t...@databricks.com wrote:
Your streaming job may have been seemingly running ok, but the DStream
checkpointing must have been
Spark 1.4.1 just got released! So just download that. Yay for timing.
On Wed, Jul 15, 2015 at 2:47 PM, Ted Yu yuzhih...@gmail.com wrote:
Should be this one:
[SPARK-7180] [SPARK-8090] [SPARK-8091] Fix a number of
SerializationDebugger bugs and limitations
...
Closes #6625 from
Should be this one:
[SPARK-7180] [SPARK-8090] [SPARK-8091] Fix a number of
SerializationDebugger bugs and limitations
...
Closes #6625 from tdas/SPARK-7180 and squashes the following commits:
On Wed, Jul 15, 2015 at 2:37 PM, Chen Song chen.song...@gmail.com wrote:
Thanks
Can you point
