Hi Nicholas,
Appreciated your response.
Understand your articulated point & I will implement and let you know the
status of the problem.
Sample:
// these lines are equivalent in Spark 2.0
spark.read.format("csv").option("header", "true").load("../Downloads/*.csv")
spark.read.option("header", "true").csv("../Downloads/*.csv") Thanks & Best
Regards,
Palash Gupta
From: Nicholas Hakobian <nicholas.hakob...@rallyhealth.com>
To: "spline_pal...@yahoo.com" <spline_pal...@yahoo.com>
Cc: Marco Mistroni <mmistr...@gmail.com>; User <user@spark.apache.org>
Sent: Thursday, December 29, 2016 10:39 PM
Subject: Re: [TorrentBroadcast] Pyspark Application terminated saying "Failed
to get broadcast_1_ piece0 of broadcast_1 in Spark 2.0.0"
If you are using spark 2.0 (as listed in the stackoverflow post) why are you
using the external CSV module from Databricks? Spark 2.0 includes the
functionality from this external module natively, and its possible you are
mixing an older library with a newer spark which could explain a crash.
Nicholas Szandor Hakobian, Ph.D.Senior Data ScientistRally
healthnicholas.hakob...@rallyhealth.com
On Thu, Dec 29, 2016 at 4:00 AM, Palash Gupta <spline_pal...@yahoo.com.invalid>
wrote:
Hi Marco,
Thanks for your response.
Yes I tested it before & am able to load from linux filesystem and it also
sometimes have similar issue.
However in both cases (either from hadoop or linux file system), this error
comes in some specific scenario as per my observations:
1. When two parallel spark separate application is initiated from one driver
(not all the time, sometime)2. If one spark jobs are running for more than
expected hour let say 2-3 hours, the second application terminated giving the
error.
To debug the problem for me it will be good if you can share some possible
reasons why failed to broadcast error may come.
Or if you need more logs I can share.
Thanks again Spark User Group.
Best RegardsPalash Gupta
Sent from Yahoo Mail on Android
On Thu, 29 Dec, 2016 at 2:57 pm, Marco Mistroni<mmistr...@gmail.com> wrote:
Hi Pls try to read a CSV from filesystem instead of hadoop. If you can read it
successfully then your hadoop file is the issue and you can start debugging
from there.Hth
On 29 Dec 2016 6:26 am, "Palash Gupta" <spline_pal...@yahoo.com. invalid> wrote:
Hi Apache Spark User team,
Greetings!
I started developing an application using Apache Hadoop and Spark using python.
My pyspark application randomly terminated saying "Failed to get broadcast_1*"
and I have been searching for suggestion and support in Stakeoverflow at Failed
to get broadcast_1_piece0 of broadcast_1 in pyspark application
|
|
|
| | |
|
|
|
| |
Failed to get broadcast_1_piece0 of broadcast_1 in pyspark application
I was building an application on Apache Spark 2.00 with Python 3.4 and trying
to load some CSV files from HDFS (... | |
|
|
Could you please provide suggestion registering myself in Apache User list or
how can I get suggestion or support to debug the problem I am facing?
Your response will be highly appreciated.
Thanks & Best Regards,
Engr. Palash GuptaWhatsApp/Viber: +8801817181502Skype: palash2494
