Ok Marcelo,
Thanks for the quick and thorough replies. I’ll keep an eye on these tickets
and the mailing list to see how things move along.
mn
On Aug 20, 2014, at 1:33 PM, Marcelo Vanzin wrote:
> Hi,
>
> On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell wrote:
>> Specifying the driver-class-p
Hi,
On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell wrote:
> Specifying the driver-class-path yields behavior like
> https://issues.apache.org/jira/browse/SPARK-2420 and
> https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a
> can of worms here if I also need to replace the gu
Marcelo,
Specifying the driver-class-path yields behavior like
https://issues.apache.org/jira/browse/SPARK-2420 and
https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a can
of worms here if I also need to replace the guava dependencies.
Wouldn’t calling “./make-distributio
Ah, sorry, forgot to talk about the second issue.
On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell wrote:
> However, now the Spark jobs running in the ApplicationMaster on a given node
> fails to find the active resourcemanager. Below is a log excerpt from one
> of the assigned nodes. As all the j
On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell wrote:
> An “unaccepted” reply to this thread from Dean Chen suggested to build Spark
> with a newer version of Hadoop (2.4.1) and this has worked to some extent.
> I’m now able to submit jobs (omitting an explicit
> “yarn.resourcemanager.address” prop
Yes, I’m pretty sure my YARN and HDFS HA configuration is correct. I can use
the UIs and HDFS command line tools with HA support as expected (failing over
namenodes and resourcemanagers, etc) so I believe this to be a Spark issue.
Like I mentioned earlier, if i manipulate the “yarn.resourcemana
Hi Matt,
I checked in the YARN code and I don't see any references to
yarn.resourcemanager.address. Have you made sure that your YARN client
configuration on the node you're launching from contains the right configs?
-Sandy
On Mon, Aug 18, 2014 at 4:07 PM, Matt Narrell
wrote:
> Hello,
>
> I
Hello,
I have an HA enabled YARN cluster with two resource mangers. When submitting
jobs via “spark-submit —master yarn-cluster”. It appears that the driver is
looking explicitly for the "yarn.resourcemanager.address” property rather than
round robin-ing through the resource managers via the
