Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
yep.  that's exactly the behavior i saw earlier, and will be figuring out
first thing tomorrow morning.  i bet it's an environment issues on the
slaves.


On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas  wrote:

> Looks like during the last build
> 
> Jenkins was unable to execute a git fetch?
>
>
> On Thu, Sep 4, 2014 at 7:58 PM, shane knapp  wrote:
>
>> i'm going to restart jenkins and see if that fixes things.
>>
>>
>> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp  wrote:
>>
>>> looking
>>>
>>>
>>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas <
>>> nicholas.cham...@gmail.com> wrote:
>>>
 It appears that our main man is having trouble
 
  hearing new requests
 .

 Do we need some smelling salts?


 On Thu, Sep 4, 2014 at 5:49 PM, shane knapp 
 wrote:

> i'd ping the Jenkinsmench...  the master was completely offline, so
> any new
> jobs wouldn't have reached it.  any jobs that were queued when power
> was
> lost probably started up, but jobs that were running would fail.
>
>
> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas <
> nicholas.cham...@gmail.com
> > wrote:
>
> > Woohoo! Thanks Shane.
> >
> > Do you know if queued PR builds will automatically be picked up? Or
> do we
> > have to ping the Jenkinmensch manually from each PR?
> >
> > Nick
> >
> >
> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp 
> wrote:
> >
> >> AND WE'RE UP!
> >>
> >> sorry that this took so long...  i'll send out a more detailed
> explanation
> >> of what happened soon.
> >>
> >> now, off to back up jenkins.
> >>
> >> shane
> >>
> >>
> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
> wrote:
> >>
> >> > it's a faulty power switch on the firewall, which has been
> swapped out.
> >> >  we're about to reboot and be good to go.
> >> >
> >> >
> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
> >> wrote:
> >> >
> >> >> looks like some hardware failed, and we're swapping in a
> replacement.
> >> i
> >> >> don't have more specific information yet -- including *what*
> failed,
> >> as our
> >> >> sysadmin is super busy ATM.  the root cause was an incorrect
> circuit
> >> being
> >> >> switched off during building maintenance.
> >> >>
> >> >> on a side note, this incident will be accelerating our plan to
> move the
> >> >> entire jenkins infrastructure in to a managed datacenter
> environment.
> >> this
> >> >> will be our major push over the next couple of weeks.  more
> details
> >> about
> >> >> this, also, as soon as i get them.
> >> >>
> >> >> i'm very sorry about the downtime, we'll get everything up and
> running
> >> >> ASAP.
> >> >>
> >> >>
> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp <
> skn...@berkeley.edu>
> >> wrote:
> >> >>
> >> >>> looks like a power outage in soda hall.  more updates as they
> happen.
> >> >>>
> >> >>>
> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp <
> skn...@berkeley.edu>
> >> >>> wrote:
> >> >>>
> >>  i am trying to get things up and running, but it looks like
> either
> >> the
> >>  firewall gateway or jenkins server itself is down.  i'll
> update as
> >> soon as
> >>  i know more.
> >> 
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >
> >  --
> > You received this message because you are subscribed to the Google
> Groups
> > "amp-infra" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an
> > email to amp-infra+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
>


>>>
>>
>

Re: amplab jenkins is down

2014-09-04 Thread Nicholas Chammas 
Looks like during the last build

Jenkins was unable to execute a git fetch?


On Thu, Sep 4, 2014 at 7:58 PM, shane knapp  wrote:

> i'm going to restart jenkins and see if that fixes things.
>
>
> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp  wrote:
>
>> looking
>>
>>
>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas <
>> nicholas.cham...@gmail.com> wrote:
>>
>>> It appears that our main man is having trouble
>>> 
>>>  hearing new requests
>>> .
>>>
>>> Do we need some smelling salts?
>>>
>>>
>>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp  wrote:
>>>
 i'd ping the Jenkinsmench...  the master was completely offline, so any
 new
 jobs wouldn't have reached it.  any jobs that were queued when power was
 lost probably started up, but jobs that were running would fail.


 On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas <
 nicholas.cham...@gmail.com
 > wrote:

 > Woohoo! Thanks Shane.
 >
 > Do you know if queued PR builds will automatically be picked up? Or
 do we
 > have to ping the Jenkinmensch manually from each PR?
 >
 > Nick
 >
 >
 > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp 
 wrote:
 >
 >> AND WE'RE UP!
 >>
 >> sorry that this took so long...  i'll send out a more detailed
 explanation
 >> of what happened soon.
 >>
 >> now, off to back up jenkins.
 >>
 >> shane
 >>
 >>
 >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
 wrote:
 >>
 >> > it's a faulty power switch on the firewall, which has been swapped
 out.
 >> >  we're about to reboot and be good to go.
 >> >
 >> >
 >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
 >> wrote:
 >> >
 >> >> looks like some hardware failed, and we're swapping in a
 replacement.
 >> i
 >> >> don't have more specific information yet -- including *what*
 failed,
 >> as our
 >> >> sysadmin is super busy ATM.  the root cause was an incorrect
 circuit
 >> being
 >> >> switched off during building maintenance.
 >> >>
 >> >> on a side note, this incident will be accelerating our plan to
 move the
 >> >> entire jenkins infrastructure in to a managed datacenter
 environment.
 >> this
 >> >> will be our major push over the next couple of weeks.  more
 details
 >> about
 >> >> this, also, as soon as i get them.
 >> >>
 >> >> i'm very sorry about the downtime, we'll get everything up and
 running
 >> >> ASAP.
 >> >>
 >> >>
 >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp >>> >
 >> wrote:
 >> >>
 >> >>> looks like a power outage in soda hall.  more updates as they
 happen.
 >> >>>
 >> >>>
 >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp <
 skn...@berkeley.edu>
 >> >>> wrote:
 >> >>>
 >>  i am trying to get things up and running, but it looks like
 either
 >> the
 >>  firewall gateway or jenkins server itself is down.  i'll update
 as
 >> soon as
 >>  i know more.
 >> 
 >> >>>
 >> >>>
 >> >>
 >> >
 >>
 >
 >  --
 > You received this message because you are subscribed to the Google
 Groups
 > "amp-infra" group.
 > To unsubscribe from this group and stop receiving emails from it,
 send an
 > email to amp-infra+unsubscr...@googlegroups.com.
 > For more options, visit https://groups.google.com/d/optout.
 >

>>>
>>>
>>
>

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread Kan Zhang 
+1

Compiled, ran newly-introduced PySpark Hadoop input/output examples.


On Thu, Sep 4, 2014 at 1:10 PM, Egor Pahomov  wrote:

> +1
>
> Compiled, ran on yarn-hadoop-2.3 simple job.
>
>
> 2014-09-04 22:22 GMT+04:00 Henry Saputra :
>
> > LICENSE and NOTICE files are good
> > Hash files are good
> > Signature files are good
> > No 3rd parties executables
> > Source compiled
> > Run local and standalone tests
> > Test persist off heap with Tachyon looks good
> >
> > +1
> >
> > - Henry
> >
> > On Wed, Sep 3, 2014 at 12:24 AM, Patrick Wendell 
> > wrote:
> > > Please vote on releasing the following candidate as Apache Spark
> version
> > 1.1.0!
> > >
> > > The tag to be voted on is v1.1.0-rc4 (commit 2f9b2bd):
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=2f9b2bd7844ee8393dc9c319f4fefedf95f5e460
> > >
> > > The release files, including signatures, digests, etc. can be found at:
> > > http://people.apache.org/~pwendell/spark-1.1.0-rc4/
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/pwendell.asc
> > >
> > > The staging repository for this release can be found at:
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1031/
> > >
> > > The documentation corresponding to this release can be found at:
> > > http://people.apache.org/~pwendell/spark-1.1.0-rc4-docs/
> > >
> > > Please vote on releasing this package as Apache Spark 1.1.0!
> > >
> > > The vote is open until Saturday, September 06, at 08:30 UTC and passes
> if
> > > a majority of at least 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Spark 1.1.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > To learn more about Apache Spark, please see
> > > http://spark.apache.org/
> > >
> > > == Regressions fixed since RC3 ==
> > > SPARK-3332 - Issue with tagging in EC2 scripts
> > > SPARK-3358 - Issue with regression for m3.XX instances
> > >
> > > == What justifies a -1 vote for this release? ==
> > > This vote is happening very late into the QA period compared with
> > > previous votes, so -1 votes should only occur for significant
> > > regressions from 1.0.2. Bugs already present in 1.0.X will not block
> > > this release.
> > >
> > > == What default changes should I be aware of? ==
> > > 1. The default value of "spark.io.compression.codec" is now "snappy"
> > > --> Old behavior can be restored by switching to "lzf"
> > >
> > > 2. PySpark now performs external spilling during aggregations.
> > > --> Old behavior can be restored by setting "spark.shuffle.spill" to
> > "false".
> > >
> > > 3. PySpark uses a new heuristic for determining the parallelism of
> > > shuffle operations.
> > > --> Old behavior can be restored by setting
> > > "spark.default.parallelism" to the number of cores in the cluster.
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> > > For additional commands, e-mail: dev-h...@spark.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> > For additional commands, e-mail: dev-h...@spark.apache.org
> >
> >
>
>
> --
>
>
>
> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
i'm going to restart jenkins and see if that fixes things.


On Thu, Sep 4, 2014 at 4:56 PM, shane knapp  wrote:

> looking
>
>
> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas <
> nicholas.cham...@gmail.com> wrote:
>
>> It appears that our main man is having trouble
>> 
>>  hearing new requests
>> .
>>
>> Do we need some smelling salts?
>>
>>
>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp  wrote:
>>
>>> i'd ping the Jenkinsmench...  the master was completely offline, so any
>>> new
>>> jobs wouldn't have reached it.  any jobs that were queued when power was
>>> lost probably started up, but jobs that were running would fail.
>>>
>>>
>>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas <
>>> nicholas.cham...@gmail.com
>>> > wrote:
>>>
>>> > Woohoo! Thanks Shane.
>>> >
>>> > Do you know if queued PR builds will automatically be picked up? Or do
>>> we
>>> > have to ping the Jenkinmensch manually from each PR?
>>> >
>>> > Nick
>>> >
>>> >
>>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp 
>>> wrote:
>>> >
>>> >> AND WE'RE UP!
>>> >>
>>> >> sorry that this took so long...  i'll send out a more detailed
>>> explanation
>>> >> of what happened soon.
>>> >>
>>> >> now, off to back up jenkins.
>>> >>
>>> >> shane
>>> >>
>>> >>
>>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
>>> wrote:
>>> >>
>>> >> > it's a faulty power switch on the firewall, which has been swapped
>>> out.
>>> >> >  we're about to reboot and be good to go.
>>> >> >
>>> >> >
>>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
>>> >> wrote:
>>> >> >
>>> >> >> looks like some hardware failed, and we're swapping in a
>>> replacement.
>>> >> i
>>> >> >> don't have more specific information yet -- including *what*
>>> failed,
>>> >> as our
>>> >> >> sysadmin is super busy ATM.  the root cause was an incorrect
>>> circuit
>>> >> being
>>> >> >> switched off during building maintenance.
>>> >> >>
>>> >> >> on a side note, this incident will be accelerating our plan to
>>> move the
>>> >> >> entire jenkins infrastructure in to a managed datacenter
>>> environment.
>>> >> this
>>> >> >> will be our major push over the next couple of weeks.  more details
>>> >> about
>>> >> >> this, also, as soon as i get them.
>>> >> >>
>>> >> >> i'm very sorry about the downtime, we'll get everything up and
>>> running
>>> >> >> ASAP.
>>> >> >>
>>> >> >>
>>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp 
>>> >> wrote:
>>> >> >>
>>> >> >>> looks like a power outage in soda hall.  more updates as they
>>> happen.
>>> >> >>>
>>> >> >>>
>>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp >> >
>>> >> >>> wrote:
>>> >> >>>
>>> >>  i am trying to get things up and running, but it looks like
>>> either
>>> >> the
>>> >>  firewall gateway or jenkins server itself is down.  i'll update
>>> as
>>> >> soon as
>>> >>  i know more.
>>> >> 
>>> >> >>>
>>> >> >>>
>>> >> >>
>>> >> >
>>> >>
>>> >
>>> >  --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "amp-infra" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> > email to amp-infra+unsubscr...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/d/optout.
>>> >
>>>
>>
>>
>

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
looking


On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas  wrote:

> It appears that our main man is having trouble
> 
>  hearing new requests
> .
>
> Do we need some smelling salts?
>
>
> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp  wrote:
>
>> i'd ping the Jenkinsmench...  the master was completely offline, so any
>> new
>> jobs wouldn't have reached it.  any jobs that were queued when power was
>> lost probably started up, but jobs that were running would fail.
>>
>>
>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas <
>> nicholas.cham...@gmail.com
>> > wrote:
>>
>> > Woohoo! Thanks Shane.
>> >
>> > Do you know if queued PR builds will automatically be picked up? Or do
>> we
>> > have to ping the Jenkinmensch manually from each PR?
>> >
>> > Nick
>> >
>> >
>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp 
>> wrote:
>> >
>> >> AND WE'RE UP!
>> >>
>> >> sorry that this took so long...  i'll send out a more detailed
>> explanation
>> >> of what happened soon.
>> >>
>> >> now, off to back up jenkins.
>> >>
>> >> shane
>> >>
>> >>
>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
>> wrote:
>> >>
>> >> > it's a faulty power switch on the firewall, which has been swapped
>> out.
>> >> >  we're about to reboot and be good to go.
>> >> >
>> >> >
>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
>> >> wrote:
>> >> >
>> >> >> looks like some hardware failed, and we're swapping in a
>> replacement.
>> >> i
>> >> >> don't have more specific information yet -- including *what* failed,
>> >> as our
>> >> >> sysadmin is super busy ATM.  the root cause was an incorrect circuit
>> >> being
>> >> >> switched off during building maintenance.
>> >> >>
>> >> >> on a side note, this incident will be accelerating our plan to move
>> the
>> >> >> entire jenkins infrastructure in to a managed datacenter
>> environment.
>> >> this
>> >> >> will be our major push over the next couple of weeks.  more details
>> >> about
>> >> >> this, also, as soon as i get them.
>> >> >>
>> >> >> i'm very sorry about the downtime, we'll get everything up and
>> running
>> >> >> ASAP.
>> >> >>
>> >> >>
>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp 
>> >> wrote:
>> >> >>
>> >> >>> looks like a power outage in soda hall.  more updates as they
>> happen.
>> >> >>>
>> >> >>>
>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp 
>> >> >>> wrote:
>> >> >>>
>> >>  i am trying to get things up and running, but it looks like either
>> >> the
>> >>  firewall gateway or jenkins server itself is down.  i'll update as
>> >> soon as
>> >>  i know more.
>> >> 
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >>
>> >
>> >  --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "amp-infra" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to amp-infra+unsubscr...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>> >
>>
>
>

Re: amplab jenkins is down

2014-09-04 Thread Patrick Wendell 
Hm yeah it seems that it hasn't been polling since 3:45.

On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas
 wrote:
> It appears that our main man is having trouble hearing new requests.
>
> Do we need some smelling salts?
>
>
> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp  wrote:
>>
>> i'd ping the Jenkinsmench...  the master was completely offline, so any
>> new
>> jobs wouldn't have reached it.  any jobs that were queued when power was
>> lost probably started up, but jobs that were running would fail.
>>
>>
>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas
>> > > wrote:
>>
>> > Woohoo! Thanks Shane.
>> >
>> > Do you know if queued PR builds will automatically be picked up? Or do
>> > we
>> > have to ping the Jenkinmensch manually from each PR?
>> >
>> > Nick
>> >
>> >
>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp  wrote:
>> >
>> >> AND WE'RE UP!
>> >>
>> >> sorry that this took so long...  i'll send out a more detailed
>> >> explanation
>> >> of what happened soon.
>> >>
>> >> now, off to back up jenkins.
>> >>
>> >> shane
>> >>
>> >>
>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
>> >> wrote:
>> >>
>> >> > it's a faulty power switch on the firewall, which has been swapped
>> >> > out.
>> >> >  we're about to reboot and be good to go.
>> >> >
>> >> >
>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
>> >> wrote:
>> >> >
>> >> >> looks like some hardware failed, and we're swapping in a
>> >> >> replacement.
>> >> i
>> >> >> don't have more specific information yet -- including *what* failed,
>> >> as our
>> >> >> sysadmin is super busy ATM.  the root cause was an incorrect circuit
>> >> being
>> >> >> switched off during building maintenance.
>> >> >>
>> >> >> on a side note, this incident will be accelerating our plan to move
>> >> >> the
>> >> >> entire jenkins infrastructure in to a managed datacenter
>> >> >> environment.
>> >> this
>> >> >> will be our major push over the next couple of weeks.  more details
>> >> about
>> >> >> this, also, as soon as i get them.
>> >> >>
>> >> >> i'm very sorry about the downtime, we'll get everything up and
>> >> >> running
>> >> >> ASAP.
>> >> >>
>> >> >>
>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp 
>> >> wrote:
>> >> >>
>> >> >>> looks like a power outage in soda hall.  more updates as they
>> >> >>> happen.
>> >> >>>
>> >> >>>
>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp 
>> >> >>> wrote:
>> >> >>>
>> >>  i am trying to get things up and running, but it looks like either
>> >> the
>> >>  firewall gateway or jenkins server itself is down.  i'll update as
>> >> soon as
>> >>  i know more.
>> >> 
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >>
>> >
>> >  --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "amp-infra" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to amp-infra+unsubscr...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>> >
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "amp-infra" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to amp-infra+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: amplab jenkins is down

2014-09-04 Thread Nicholas Chammas 
It appears that our main man is having trouble

 hearing new requests
.

Do we need some smelling salts?


On Thu, Sep 4, 2014 at 5:49 PM, shane knapp  wrote:

> i'd ping the Jenkinsmench...  the master was completely offline, so any new
> jobs wouldn't have reached it.  any jobs that were queued when power was
> lost probably started up, but jobs that were running would fail.
>
>
> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas <
> nicholas.cham...@gmail.com
> > wrote:
>
> > Woohoo! Thanks Shane.
> >
> > Do you know if queued PR builds will automatically be picked up? Or do we
> > have to ping the Jenkinmensch manually from each PR?
> >
> > Nick
> >
> >
> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp  wrote:
> >
> >> AND WE'RE UP!
> >>
> >> sorry that this took so long...  i'll send out a more detailed
> explanation
> >> of what happened soon.
> >>
> >> now, off to back up jenkins.
> >>
> >> shane
> >>
> >>
> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
> wrote:
> >>
> >> > it's a faulty power switch on the firewall, which has been swapped
> out.
> >> >  we're about to reboot and be good to go.
> >> >
> >> >
> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
> >> wrote:
> >> >
> >> >> looks like some hardware failed, and we're swapping in a replacement.
> >> i
> >> >> don't have more specific information yet -- including *what* failed,
> >> as our
> >> >> sysadmin is super busy ATM.  the root cause was an incorrect circuit
> >> being
> >> >> switched off during building maintenance.
> >> >>
> >> >> on a side note, this incident will be accelerating our plan to move
> the
> >> >> entire jenkins infrastructure in to a managed datacenter environment.
> >> this
> >> >> will be our major push over the next couple of weeks.  more details
> >> about
> >> >> this, also, as soon as i get them.
> >> >>
> >> >> i'm very sorry about the downtime, we'll get everything up and
> running
> >> >> ASAP.
> >> >>
> >> >>
> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp 
> >> wrote:
> >> >>
> >> >>> looks like a power outage in soda hall.  more updates as they
> happen.
> >> >>>
> >> >>>
> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp 
> >> >>> wrote:
> >> >>>
> >>  i am trying to get things up and running, but it looks like either
> >> the
> >>  firewall gateway or jenkins server itself is down.  i'll update as
> >> soon as
> >>  i know more.
> >> 
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "amp-infra" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to amp-infra+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
>

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
i'd ping the Jenkinsmench...  the master was completely offline, so any new
jobs wouldn't have reached it.  any jobs that were queued when power was
lost probably started up, but jobs that were running would fail.


On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas  wrote:

> Woohoo! Thanks Shane.
>
> Do you know if queued PR builds will automatically be picked up? Or do we
> have to ping the Jenkinmensch manually from each PR?
>
> Nick
>
>
> On Thu, Sep 4, 2014 at 5:37 PM, shane knapp  wrote:
>
>> AND WE'RE UP!
>>
>> sorry that this took so long...  i'll send out a more detailed explanation
>> of what happened soon.
>>
>> now, off to back up jenkins.
>>
>> shane
>>
>>
>> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp  wrote:
>>
>> > it's a faulty power switch on the firewall, which has been swapped out.
>> >  we're about to reboot and be good to go.
>> >
>> >
>> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp 
>> wrote:
>> >
>> >> looks like some hardware failed, and we're swapping in a replacement.
>> i
>> >> don't have more specific information yet -- including *what* failed,
>> as our
>> >> sysadmin is super busy ATM.  the root cause was an incorrect circuit
>> being
>> >> switched off during building maintenance.
>> >>
>> >> on a side note, this incident will be accelerating our plan to move the
>> >> entire jenkins infrastructure in to a managed datacenter environment.
>> this
>> >> will be our major push over the next couple of weeks.  more details
>> about
>> >> this, also, as soon as i get them.
>> >>
>> >> i'm very sorry about the downtime, we'll get everything up and running
>> >> ASAP.
>> >>
>> >>
>> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp 
>> wrote:
>> >>
>> >>> looks like a power outage in soda hall.  more updates as they happen.
>> >>>
>> >>>
>> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp 
>> >>> wrote:
>> >>>
>>  i am trying to get things up and running, but it looks like either
>> the
>>  firewall gateway or jenkins server itself is down.  i'll update as
>> soon as
>>  i know more.
>> 
>> >>>
>> >>>
>> >>
>> >
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "amp-infra" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to amp-infra+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

Re: amplab jenkins is down

2014-09-04 Thread Nicholas Chammas 
Woohoo! Thanks Shane.

Do you know if queued PR builds will automatically be picked up? Or do we
have to ping the Jenkinmensch manually from each PR?

Nick


On Thu, Sep 4, 2014 at 5:37 PM, shane knapp  wrote:

> AND WE'RE UP!
>
> sorry that this took so long...  i'll send out a more detailed explanation
> of what happened soon.
>
> now, off to back up jenkins.
>
> shane
>
>
> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp  wrote:
>
> > it's a faulty power switch on the firewall, which has been swapped out.
> >  we're about to reboot and be good to go.
> >
> >
> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp  wrote:
> >
> >> looks like some hardware failed, and we're swapping in a replacement.  i
> >> don't have more specific information yet -- including *what* failed, as
> our
> >> sysadmin is super busy ATM.  the root cause was an incorrect circuit
> being
> >> switched off during building maintenance.
> >>
> >> on a side note, this incident will be accelerating our plan to move the
> >> entire jenkins infrastructure in to a managed datacenter environment.
> this
> >> will be our major push over the next couple of weeks.  more details
> about
> >> this, also, as soon as i get them.
> >>
> >> i'm very sorry about the downtime, we'll get everything up and running
> >> ASAP.
> >>
> >>
> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp 
> wrote:
> >>
> >>> looks like a power outage in soda hall.  more updates as they happen.
> >>>
> >>>
> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp 
> >>> wrote:
> >>>
>  i am trying to get things up and running, but it looks like either the
>  firewall gateway or jenkins server itself is down.  i'll update as
> soon as
>  i know more.
> 
> >>>
> >>>
> >>
> >
>

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
AND WE'RE UP!

sorry that this took so long...  i'll send out a more detailed explanation
of what happened soon.

now, off to back up jenkins.

shane


On Thu, Sep 4, 2014 at 1:27 PM, shane knapp  wrote:

> it's a faulty power switch on the firewall, which has been swapped out.
>  we're about to reboot and be good to go.
>
>
> On Thu, Sep 4, 2014 at 1:19 PM, shane knapp  wrote:
>
>> looks like some hardware failed, and we're swapping in a replacement.  i
>> don't have more specific information yet -- including *what* failed, as our
>> sysadmin is super busy ATM.  the root cause was an incorrect circuit being
>> switched off during building maintenance.
>>
>> on a side note, this incident will be accelerating our plan to move the
>> entire jenkins infrastructure in to a managed datacenter environment.  this
>> will be our major push over the next couple of weeks.  more details about
>> this, also, as soon as i get them.
>>
>> i'm very sorry about the downtime, we'll get everything up and running
>> ASAP.
>>
>>
>> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp  wrote:
>>
>>> looks like a power outage in soda hall.  more updates as they happen.
>>>
>>>
>>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp 
>>> wrote:
>>>
 i am trying to get things up and running, but it looks like either the
 firewall gateway or jenkins server itself is down.  i'll update as soon as
 i know more.

>>>
>>>
>>
>

How to kill a Spark job running in local mode programmatically ?

2014-09-04 Thread randomuser54 
I have a java class which calls SparkSubmit.scala with all the arguments to
run a spark job in a thread. I am running them in local mode for now but
also want to run them in yarn-cluster mode later.

Now, I want to kill the running spark job (which can be in local or
yarn-cluster mode) programmatically.

I know that SparkContext has a stop() method but from the thread from which
I am calling the SparkSubmit I don’t have access to it. Can someone suggest
me how to do this properly ?

Thanks.




--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-kill-a-Spark-job-running-in-local-mode-programmatically-tp8279.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread randomuser54 
+1



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-1-0-RC4-tp8219p8278.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread Nicholas Chammas 
On Thu, Sep 4, 2014 at 1:50 PM, Gurvinder Singh 
wrote:

> There is a regression when using pyspark to read data
> from HDFS.
>

Could you open a JIRA  with a brief repro?
We'll look into it.

(You could also provide a repro in a separate thread.)

Nick

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
it's a faulty power switch on the firewall, which has been swapped out.
 we're about to reboot and be good to go.


On Thu, Sep 4, 2014 at 1:19 PM, shane knapp  wrote:

> looks like some hardware failed, and we're swapping in a replacement.  i
> don't have more specific information yet -- including *what* failed, as our
> sysadmin is super busy ATM.  the root cause was an incorrect circuit being
> switched off during building maintenance.
>
> on a side note, this incident will be accelerating our plan to move the
> entire jenkins infrastructure in to a managed datacenter environment.  this
> will be our major push over the next couple of weeks.  more details about
> this, also, as soon as i get them.
>
> i'm very sorry about the downtime, we'll get everything up and running
> ASAP.
>
>
> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp  wrote:
>
>> looks like a power outage in soda hall.  more updates as they happen.
>>
>>
>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp  wrote:
>>
>>> i am trying to get things up and running, but it looks like either the
>>> firewall gateway or jenkins server itself is down.  i'll update as soon as
>>> i know more.
>>>
>>
>>
>

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
looks like some hardware failed, and we're swapping in a replacement.  i
don't have more specific information yet -- including *what* failed, as our
sysadmin is super busy ATM.  the root cause was an incorrect circuit being
switched off during building maintenance.

on a side note, this incident will be accelerating our plan to move the
entire jenkins infrastructure in to a managed datacenter environment.  this
will be our major push over the next couple of weeks.  more details about
this, also, as soon as i get them.

i'm very sorry about the downtime, we'll get everything up and running ASAP.


On Thu, Sep 4, 2014 at 12:27 PM, shane knapp  wrote:

> looks like a power outage in soda hall.  more updates as they happen.
>
>
> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp  wrote:
>
>> i am trying to get things up and running, but it looks like either the
>> firewall gateway or jenkins server itself is down.  i'll update as soon as
>> i know more.
>>
>
>

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread Egor Pahomov 
+1

Compiled, ran on yarn-hadoop-2.3 simple job.


2014-09-04 22:22 GMT+04:00 Henry Saputra :

> LICENSE and NOTICE files are good
> Hash files are good
> Signature files are good
> No 3rd parties executables
> Source compiled
> Run local and standalone tests
> Test persist off heap with Tachyon looks good
>
> +1
>
> - Henry
>
> On Wed, Sep 3, 2014 at 12:24 AM, Patrick Wendell 
> wrote:
> > Please vote on releasing the following candidate as Apache Spark version
> 1.1.0!
> >
> > The tag to be voted on is v1.1.0-rc4 (commit 2f9b2bd):
> >
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=2f9b2bd7844ee8393dc9c319f4fefedf95f5e460
> >
> > The release files, including signatures, digests, etc. can be found at:
> > http://people.apache.org/~pwendell/spark-1.1.0-rc4/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pwendell.asc
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1031/
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~pwendell/spark-1.1.0-rc4-docs/
> >
> > Please vote on releasing this package as Apache Spark 1.1.0!
> >
> > The vote is open until Saturday, September 06, at 08:30 UTC and passes if
> > a majority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Spark 1.1.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> > http://spark.apache.org/
> >
> > == Regressions fixed since RC3 ==
> > SPARK-3332 - Issue with tagging in EC2 scripts
> > SPARK-3358 - Issue with regression for m3.XX instances
> >
> > == What justifies a -1 vote for this release? ==
> > This vote is happening very late into the QA period compared with
> > previous votes, so -1 votes should only occur for significant
> > regressions from 1.0.2. Bugs already present in 1.0.X will not block
> > this release.
> >
> > == What default changes should I be aware of? ==
> > 1. The default value of "spark.io.compression.codec" is now "snappy"
> > --> Old behavior can be restored by switching to "lzf"
> >
> > 2. PySpark now performs external spilling during aggregations.
> > --> Old behavior can be restored by setting "spark.shuffle.spill" to
> "false".
> >
> > 3. PySpark uses a new heuristic for determining the parallelism of
> > shuffle operations.
> > --> Old behavior can be restored by setting
> > "spark.default.parallelism" to the number of cores in the cluster.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> > For additional commands, e-mail: dev-h...@spark.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>


-- 



*Sincerely yoursEgor PakhomovScala Developer, Yandex*

Re: amplab jenkins is down

2014-09-04 Thread shane knapp 
looks like a power outage in soda hall.  more updates as they happen.


On Thu, Sep 4, 2014 at 12:25 PM, shane knapp  wrote:

> i am trying to get things up and running, but it looks like either the
> firewall gateway or jenkins server itself is down.  i'll update as soon as
> i know more.
>

amplab jenkins is down

2014-09-04 Thread shane knapp 
i am trying to get things up and running, but it looks like either the
firewall gateway or jenkins server itself is down.  i'll update as soon as
i know more.

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread Henry Saputra 
LICENSE and NOTICE files are good
Hash files are good
Signature files are good
No 3rd parties executables
Source compiled
Run local and standalone tests
Test persist off heap with Tachyon looks good

+1

- Henry

On Wed, Sep 3, 2014 at 12:24 AM, Patrick Wendell  wrote:
> Please vote on releasing the following candidate as Apache Spark version 
> 1.1.0!
>
> The tag to be voted on is v1.1.0-rc4 (commit 2f9b2bd):
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=2f9b2bd7844ee8393dc9c319f4fefedf95f5e460
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-1.1.0-rc4/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1031/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-1.1.0-rc4-docs/
>
> Please vote on releasing this package as Apache Spark 1.1.0!
>
> The vote is open until Saturday, September 06, at 08:30 UTC and passes if
> a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.1.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.apache.org/
>
> == Regressions fixed since RC3 ==
> SPARK-3332 - Issue with tagging in EC2 scripts
> SPARK-3358 - Issue with regression for m3.XX instances
>
> == What justifies a -1 vote for this release? ==
> This vote is happening very late into the QA period compared with
> previous votes, so -1 votes should only occur for significant
> regressions from 1.0.2. Bugs already present in 1.0.X will not block
> this release.
>
> == What default changes should I be aware of? ==
> 1. The default value of "spark.io.compression.codec" is now "snappy"
> --> Old behavior can be restored by switching to "lzf"
>
> 2. PySpark now performs external spilling during aggregations.
> --> Old behavior can be restored by setting "spark.shuffle.spill" to "false".
>
> 3. PySpark uses a new heuristic for determining the parallelism of
> shuffle operations.
> --> Old behavior can be restored by setting
> "spark.default.parallelism" to the number of cores in the cluster.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread Gurvinder Singh 
On 09/03/2014 04:23 PM, Nicholas Chammas wrote:
> On Wed, Sep 3, 2014 at 3:24 AM, Patrick Wendell  wrote:
> 
>> == What default changes should I be aware of? ==
>> 1. The default value of "spark.io.compression.codec" is now "snappy"
>> --> Old behavior can be restored by switching to "lzf"
>>
>> 2. PySpark now performs external spilling during aggregations.
>> --> Old behavior can be restored by setting "spark.shuffle.spill" to
>> "false".
>>
>> 3. PySpark uses a new heuristic for determining the parallelism of
>> shuffle operations.
>> --> Old behavior can be restored by setting
>> "spark.default.parallelism" to the number of cores in the cluster.
>>
> 
> Will these changes be called out in the release notes or somewhere in the
> docs?
> 
> That last one (which I believe is what we discovered as the result of
> SPARK- ) could have a
> large impact on PySpark users.

Just wanted to add, it might be related to this issue or different.
There is a regression when using pyspark to read data
from HDFS. its performance during map tasks has gone down approx 1 ->
0.5x. I have tested the 1.0.2 and the performance was fine, but the 1.1
release candidate has this issue. I tested by setting the following
properties to make sure it was not due to these.

set("spark.io.compression.codec","lzf").set("spark.shuffle.spill","false")

in conf object.

Regards,
Gurvinder
> 
> Nick
> 


-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

2014-09-04 Thread Tom Graves 
+1. Ran spark on yarn on hadoop 0.23 and 2.x.

Tom


On Wednesday, September 3, 2014 2:25 AM, Patrick Wendell  
wrote:
 


Please vote on releasing the following candidate as Apache Spark version 1.1.0!

The tag to be voted on is v1.1.0-rc4 (commit 2f9b2bd):
https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=2f9b2bd7844ee8393dc9c319f4fefedf95f5e460

The release files, including signatures, digests, etc. can be found at:
http://people.apache.org/~pwendell/spark-1.1.0-rc4/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1031/

The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-1.1.0-rc4-docs/

Please vote on releasing this package as Apache Spark 1.1.0!

The vote is open until Saturday, September 06, at 08:30 UTC and passes if
a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 1.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see
http://spark.apache.org/

== Regressions fixed since RC3 ==
SPARK-3332 - Issue with tagging in EC2 scripts
SPARK-3358 - Issue with regression for m3.XX instances

== What justifies a -1 vote for this release? ==
This vote is happening very late into the QA period compared with
previous votes, so -1 votes should only occur for significant
regressions from 1.0.2. Bugs already present in 1.0.X will not block
this release.

== What
 default changes should I be aware of? ==
1. The default value of "spark.io.compression.codec" is now "snappy"
--> Old behavior can be restored by switching to "lzf"

2. PySpark now performs external spilling during aggregations.
--> Old behavior can be restored by setting "spark.shuffle.spill" to "false".

3. PySpark uses a new heuristic for determining the parallelism of
shuffle operations.
--> Old behavior can be restored by setting
"spark.default.parallelism" to the number of cores in the cluster.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

RE: Is breeze thread safe in Spark?

2014-09-04 Thread Ulanov, Alexander 
I've experienced something related to what we discussed. NaïveBayes crashes 
with native blas/lapack libraries for breeze/netlib on Windows: 
https://issues.apache.org/jira/browse/SPARK-3403
I've also attached to the issue another example with gradient that crashes in 
runMiniBatchSGD, probably trying to do grad1 += grad2.
Could you take a close look at this issue? It paralyzed my development for 
mllib...

Best regards, Alexander

-Original Message-
From: Xiangrui Meng [mailto:men...@gmail.com] 
Sent: Wednesday, September 03, 2014 11:18 PM
To: RJ Nowling
Cc: David Hall; Ulanov, Alexander; 
Subject: Re: Is breeze thread safe in Spark?

RJ, could you provide a code example that can re-produce the bug you observed 
in local testing? Breeze's += is not thread-safe. But in a Spark job, calls to 
a resultHandler is synchronized:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/JobWaiter.scala#L52
. Let's move our discussion to the JIRA page. -Xiangrui

On Wed, Sep 3, 2014 at 12:07 PM, RJ Nowling  wrote:
> Here's the JIRA:
>
> https://issues.apache.org/jira/browse/SPARK-3384
>
> Even if the current implementation uses += in a thread safe manner, it 
> can be easy to make the mistake of accidentally using += in a 
> parallelized context.  I suggest changing all instances of += to +.
>
> I would encourage others to reproduce and validate this issue, though.
>
>
> On Wed, Sep 3, 2014 at 3:02 PM, David Hall  wrote:
>
>> mutating operations are not thread safe. Operations that don't mutate 
>> should be thread safe. I can't speak to what Evan said, but I would 
>> guess that the way they're using += should be safe.
>>
>>
>> On Wed, Sep 3, 2014 at 11:58 AM, RJ Nowling  wrote:
>>
>>> David,
>>>
>>> Can you confirm that += is not thread safe but + is?  I'm assuming + 
>>> allocates a new object for the write, while += doesn't.
>>>
>>> Thanks!
>>> RJ
>>>
>>>
>>> On Wed, Sep 3, 2014 at 2:50 PM, David Hall  wrote:
>>>
 In general, in Breeze we allocate separate work arrays for each 
 call to lapack, so it should be fine. In general concurrent 
 modification isn't thread safe of course, but things that "ought" 
 to be thread safe really should be.


 On Wed, Sep 3, 2014 at 10:41 AM, RJ Nowling  wrote:

> No, it's not in all cases.   Since Breeze uses lapack under the hood,
> changes to memory between different threads is bad.
>
> There's actually a potential bug in the KMeans code where it uses 
> += instead of +.
>
>
> On Wed, Sep 3, 2014 at 1:26 PM, Ulanov, Alexander < 
> alexander.ula...@hp.com>
> wrote:
>
> > Hi,
> >
> > Is breeze library called thread safe from Spark mllib code in 
> > case
> when
> > native libs for blas and lapack are used? Might it be an issue 
> > when
> running
> > Spark locally?
> >
> > Best regards, Alexander
> > 
> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> > For additional commands, e-mail: dev-h...@spark.apache.org
> >
> >
>
>
> --
> em rnowl...@gmail.com
> c 954.496.2314
>


>>>
>>>
>>> --
>>> em rnowl...@gmail.com
>>> c 954.496.2314
>>>
>>
>>
>
>
> --
> em rnowl...@gmail.com
> c 954.496.2314

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Dependency hell in Spark applications

2014-09-04 Thread Koert Kuipers 
custom spark builds should not be the answer. at least not if spark ever
wants to have a vibrant community for spark apps.

spark does support a user-classpath-first option, which would deal with
some of these issues, but I don't think it works.
On Sep 4, 2014 9:01 AM, "Felix Garcia Borrego"  wrote:

> Hi,
> I run into the same issue and apart from the ideas Aniket said, I only
> could find a nasty workaround. Add my custom PoolingClientConnectionManager
> to my classpath.
>
>
> http://stackoverflow.com/questions/24788949/nosuchmethoderror-while-running-aws-s3-client-on-spark-while-javap-shows-otherwi/25488955#25488955
>
>
>
> On Thu, Sep 4, 2014 at 11:43 AM, Sean Owen  wrote:
>
> > Dumb question -- are you using a Spark build that includes the Kinesis
> > dependency? that build would have resolved conflicts like this for
> > you. Your app would need to use the same version of the Kinesis client
> > SDK, ideally.
> >
> > All of these ideas are well-known, yes. In cases of super-common
> > dependencies like Guava, they are already shaded. This is a
> > less-common source of conflicts so I don't think http-client is
> > shaded, especially since it is not used directly by Spark. I think
> > this is a case of your app conflicting with a third-party dependency?
> >
> > I think OSGi is deemed too over the top for things like this.
> >
> > On Thu, Sep 4, 2014 at 11:35 AM, Aniket Bhatnagar
> >  wrote:
> > > I am trying to use Kinesis as source to Spark Streaming and have run
> > into a
> > > dependency issue that can't be resolved without making my own custom
> > Spark
> > > build. The issue is that Spark is transitively dependent
> > > on org.apache.httpcomponents:httpclient:jar:4.1.2 (I think because of
> > > libfb303 coming from hbase and hive-serde) whereas AWS SDK is dependent
> > > on org.apache.httpcomponents:httpclient:jar:4.2. When I package and run
> > > Spark Streaming application, I get the following:
> > >
> > > Caused by: java.lang.NoSuchMethodError:
> > >
> >
> org.apache.http.impl.conn.DefaultClientConnectionOperator.(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
> > > at
> > >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
> > > at
> > >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:114)
> > > at
> > >
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:99)
> > > at
> > >
> >
> com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
> > > at
> > >
> >
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
> > > at
> > > com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:181)
> > > at
> > >
> >
> com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:119)
> > > at
> > >
> >
> com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:103)
> > > at
> > >
> >
> com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:136)
> > > at
> > >
> >
> com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:117)
> > > at
> > >
> >
> com.amazonaws.services.kinesis.AmazonKinesisAsyncClient.(AmazonKinesisAsyncClient.java:132)
> > >
> > > I can create a custom Spark build with
> > > org.apache.httpcomponents:httpclient:jar:4.2 included in the assembly
> > but I
> > > was wondering if this is something Spark devs have noticed and are
> > looking
> > > to resolve in near releases. Here are my thoughts on this issue:
> > >
> > > Containers that allow running custom user code have to often resolve
> > > dependency issues in case of conflicts between framework's and user
> > code's
> > > dependency. Here is how I have seen some frameworks resolve the issue:
> > > 1. Provide a child-first class loader: Some JEE containers provided a
> > > child-first class loader that allowed for loading classes from user
> code
> > > first. I don't think this approach completely solves the problem as the
> > > framework is then susceptible to class mismatch errors.
> > > 2. Fold in all dependencies in a sub-package: This approach involves
> > > folding all dependencies in a project specific sub-package (like
> > > spark.dependencies). This approach is tedious because it involves
> > building
> > > custom version of all dependencies (and their transitive dependencies)
> > > 3. Use something like OSGi: Some frameworks has successfully used OSGi
> to
> > > manage dependencies between the modules. The challenge in this approach
> > is
> > > to OSGify the framework and hide OSGi complexities from end user.
> > >
> > > My personal preference is OSGi (or atleast some support for OSGi) but I
> > > would love to hear what Spark devs are thinking in terms of resolving
> the
> > > problem.
> > >
> >

Re: Dependency hell in Spark applications

2014-09-04 Thread Felix Garcia Borrego 
Hi,
I run into the same issue and apart from the ideas Aniket said, I only
could find a nasty workaround. Add my custom PoolingClientConnectionManager
to my classpath.

http://stackoverflow.com/questions/24788949/nosuchmethoderror-while-running-aws-s3-client-on-spark-while-javap-shows-otherwi/25488955#25488955



On Thu, Sep 4, 2014 at 11:43 AM, Sean Owen  wrote:

> Dumb question -- are you using a Spark build that includes the Kinesis
> dependency? that build would have resolved conflicts like this for
> you. Your app would need to use the same version of the Kinesis client
> SDK, ideally.
>
> All of these ideas are well-known, yes. In cases of super-common
> dependencies like Guava, they are already shaded. This is a
> less-common source of conflicts so I don't think http-client is
> shaded, especially since it is not used directly by Spark. I think
> this is a case of your app conflicting with a third-party dependency?
>
> I think OSGi is deemed too over the top for things like this.
>
> On Thu, Sep 4, 2014 at 11:35 AM, Aniket Bhatnagar
>  wrote:
> > I am trying to use Kinesis as source to Spark Streaming and have run
> into a
> > dependency issue that can't be resolved without making my own custom
> Spark
> > build. The issue is that Spark is transitively dependent
> > on org.apache.httpcomponents:httpclient:jar:4.1.2 (I think because of
> > libfb303 coming from hbase and hive-serde) whereas AWS SDK is dependent
> > on org.apache.httpcomponents:httpclient:jar:4.2. When I package and run
> > Spark Streaming application, I get the following:
> >
> > Caused by: java.lang.NoSuchMethodError:
> >
> org.apache.http.impl.conn.DefaultClientConnectionOperator.(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
> > at
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
> > at
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:114)
> > at
> >
> org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:99)
> > at
> >
> com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
> > at
> >
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
> > at
> > com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:181)
> > at
> >
> com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:119)
> > at
> >
> com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:103)
> > at
> >
> com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:136)
> > at
> >
> com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:117)
> > at
> >
> com.amazonaws.services.kinesis.AmazonKinesisAsyncClient.(AmazonKinesisAsyncClient.java:132)
> >
> > I can create a custom Spark build with
> > org.apache.httpcomponents:httpclient:jar:4.2 included in the assembly
> but I
> > was wondering if this is something Spark devs have noticed and are
> looking
> > to resolve in near releases. Here are my thoughts on this issue:
> >
> > Containers that allow running custom user code have to often resolve
> > dependency issues in case of conflicts between framework's and user
> code's
> > dependency. Here is how I have seen some frameworks resolve the issue:
> > 1. Provide a child-first class loader: Some JEE containers provided a
> > child-first class loader that allowed for loading classes from user code
> > first. I don't think this approach completely solves the problem as the
> > framework is then susceptible to class mismatch errors.
> > 2. Fold in all dependencies in a sub-package: This approach involves
> > folding all dependencies in a project specific sub-package (like
> > spark.dependencies). This approach is tedious because it involves
> building
> > custom version of all dependencies (and their transitive dependencies)
> > 3. Use something like OSGi: Some frameworks has successfully used OSGi to
> > manage dependencies between the modules. The challenge in this approach
> is
> > to OSGify the framework and hide OSGi complexities from end user.
> >
> > My personal preference is OSGi (or atleast some support for OSGi) but I
> > would love to hear what Spark devs are thinking in terms of resolving the
> > problem.
> >
> > Thanks,
> > Aniket
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: Dependency hell in Spark applications

2014-09-04 Thread Sean Owen 
Dumb question -- are you using a Spark build that includes the Kinesis
dependency? that build would have resolved conflicts like this for
you. Your app would need to use the same version of the Kinesis client
SDK, ideally.

All of these ideas are well-known, yes. In cases of super-common
dependencies like Guava, they are already shaded. This is a
less-common source of conflicts so I don't think http-client is
shaded, especially since it is not used directly by Spark. I think
this is a case of your app conflicting with a third-party dependency?

I think OSGi is deemed too over the top for things like this.

On Thu, Sep 4, 2014 at 11:35 AM, Aniket Bhatnagar
 wrote:
> I am trying to use Kinesis as source to Spark Streaming and have run into a
> dependency issue that can't be resolved without making my own custom Spark
> build. The issue is that Spark is transitively dependent
> on org.apache.httpcomponents:httpclient:jar:4.1.2 (I think because of
> libfb303 coming from hbase and hive-serde) whereas AWS SDK is dependent
> on org.apache.httpcomponents:httpclient:jar:4.2. When I package and run
> Spark Streaming application, I get the following:
>
> Caused by: java.lang.NoSuchMethodError:
> org.apache.http.impl.conn.DefaultClientConnectionOperator.(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
> at
> org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
> at
> org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:114)
> at
> org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:99)
> at
> com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
> at
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
> at
> com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:181)
> at
> com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:119)
> at
> com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:103)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:136)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:117)
> at
> com.amazonaws.services.kinesis.AmazonKinesisAsyncClient.(AmazonKinesisAsyncClient.java:132)
>
> I can create a custom Spark build with
> org.apache.httpcomponents:httpclient:jar:4.2 included in the assembly but I
> was wondering if this is something Spark devs have noticed and are looking
> to resolve in near releases. Here are my thoughts on this issue:
>
> Containers that allow running custom user code have to often resolve
> dependency issues in case of conflicts between framework's and user code's
> dependency. Here is how I have seen some frameworks resolve the issue:
> 1. Provide a child-first class loader: Some JEE containers provided a
> child-first class loader that allowed for loading classes from user code
> first. I don't think this approach completely solves the problem as the
> framework is then susceptible to class mismatch errors.
> 2. Fold in all dependencies in a sub-package: This approach involves
> folding all dependencies in a project specific sub-package (like
> spark.dependencies). This approach is tedious because it involves building
> custom version of all dependencies (and their transitive dependencies)
> 3. Use something like OSGi: Some frameworks has successfully used OSGi to
> manage dependencies between the modules. The challenge in this approach is
> to OSGify the framework and hide OSGi complexities from end user.
>
> My personal preference is OSGi (or atleast some support for OSGi) but I
> would love to hear what Spark devs are thinking in terms of resolving the
> problem.
>
> Thanks,
> Aniket

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Dependency hell in Spark applications

2014-09-04 Thread Aniket Bhatnagar 
I am trying to use Kinesis as source to Spark Streaming and have run into a
dependency issue that can't be resolved without making my own custom Spark
build. The issue is that Spark is transitively dependent
on org.apache.httpcomponents:httpclient:jar:4.1.2 (I think because of
libfb303 coming from hbase and hive-serde) whereas AWS SDK is dependent
on org.apache.httpcomponents:httpclient:jar:4.2. When I package and run
Spark Streaming application, I get the following:

Caused by: java.lang.NoSuchMethodError:
org.apache.http.impl.conn.DefaultClientConnectionOperator.(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V
at
org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140)
at
org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:114)
at
org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:99)
at
com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:29)
at
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:97)
at
com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:181)
at
com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:119)
at
com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:103)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:136)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.(AmazonKinesisClient.java:117)
at
com.amazonaws.services.kinesis.AmazonKinesisAsyncClient.(AmazonKinesisAsyncClient.java:132)

I can create a custom Spark build with
org.apache.httpcomponents:httpclient:jar:4.2 included in the assembly but I
was wondering if this is something Spark devs have noticed and are looking
to resolve in near releases. Here are my thoughts on this issue:

Containers that allow running custom user code have to often resolve
dependency issues in case of conflicts between framework's and user code's
dependency. Here is how I have seen some frameworks resolve the issue:
1. Provide a child-first class loader: Some JEE containers provided a
child-first class loader that allowed for loading classes from user code
first. I don't think this approach completely solves the problem as the
framework is then susceptible to class mismatch errors.
2. Fold in all dependencies in a sub-package: This approach involves
folding all dependencies in a project specific sub-package (like
spark.dependencies). This approach is tedious because it involves building
custom version of all dependencies (and their transitive dependencies)
3. Use something like OSGi: Some frameworks has successfully used OSGi to
manage dependencies between the modules. The challenge in this approach is
to OSGify the framework and hide OSGi complexities from end user.

My personal preference is OSGi (or atleast some support for OSGi) but I
would love to hear what Spark devs are thinking in terms of resolving the
problem.

Thanks,
Aniket

Re: memory size for caching RDD

2014-09-04 Thread 牛兆捷 
ok. So can I use the similar logic as the block manager does when space
fills up ?


2014-09-04 15:05 GMT+08:00 Liu, Raymond :

> I think there is no public API available to do this. In this case, the
> best you can do might be unpersist some RDDs manually. The problem is that
> this is done by RDD unit, not by block unit. And then, if the storage level
> including disk level, the data on the disk will be removed too.
>
> Best Regards,
> Raymond Liu
>
> From: 牛兆捷 [mailto:nzjem...@gmail.com]
> Sent: Thursday, September 04, 2014 2:57 PM
> To: Liu, Raymond
> Cc: Patrick Wendell; u...@spark.apache.org; dev@spark.apache.org
> Subject: Re: memory size for caching RDD
>
> Oh I see.
>
> I want to implement something like this: sometimes I need to release some
> memory for other usage even when they are occupied by some RDDs (can be
> recomputed with the help of lineage when they are needed),  does spark
> provide interfaces to force it to release some memory ?
>
> 2014-09-04 14:32 GMT+08:00 Liu, Raymond :
> You don’t need to. It is not static allocated to RDD cache, it is just an
> up limit.
> If you don’t use up the memory by RDD cache, it is always available for
> other usage. except those one also controlled by some memoryFraction conf.
> e.g. spark.shuffle.memoryFraction which you also set the up limit.
>
> Best Regards,
> Raymond Liu
>
> From: 牛兆捷 [mailto:nzjem...@gmail.com]
> Sent: Thursday, September 04, 2014 2:27 PM
> To: Patrick Wendell
> Cc: u...@spark.apache.org; dev@spark.apache.org
> Subject: Re: memory size for caching RDD
>
> But is it possible to make t resizable? When we don't have many RDD to
> cache, we can give some memory to others.
>
> 2014-09-04 13:45 GMT+08:00 Patrick Wendell :
> Changing this is not supported, it si immutable similar to other spark
> configuration settings.
>
> On Wed, Sep 3, 2014 at 8:13 PM, 牛兆捷  wrote:
> > Dear all:
> >
> > Spark uses memory to cache RDD and the memory size is specified by
> > "spark.storage.memoryFraction".
> >
> > One the Executor starts, does Spark support adjusting/resizing memory
> size
> > of this part dynamically?
> >
> > Thanks.
> >
> > --
> > *Regards,*
> > *Zhaojie*
>
>
>
> --
> Regards,
> Zhaojie
>
>
>
>
> --
> Regards,
> Zhaojie
>
>


-- 
*Regards,*
*Zhaojie*

RE: memory size for caching RDD

2014-09-04 Thread Liu, Raymond 
I think there is no public API available to do this. In this case, the best you 
can do might be unpersist some RDDs manually. The problem is that this is done 
by RDD unit, not by block unit. And then, if the storage level including disk 
level, the data on the disk will be removed too.

Best Regards,
Raymond Liu

From: 牛兆捷 [mailto:nzjem...@gmail.com] 
Sent: Thursday, September 04, 2014 2:57 PM
To: Liu, Raymond
Cc: Patrick Wendell; u...@spark.apache.org; dev@spark.apache.org
Subject: Re: memory size for caching RDD

Oh I see. 

I want to implement something like this: sometimes I need to release some 
memory for other usage even when they are occupied by some RDDs (can be 
recomputed with the help of lineage when they are needed),  does spark provide 
interfaces to force it to release some memory ?

2014-09-04 14:32 GMT+08:00 Liu, Raymond :
You don’t need to. It is not static allocated to RDD cache, it is just an up 
limit.
If you don’t use up the memory by RDD cache, it is always available for other 
usage. except those one also controlled by some memoryFraction conf. e.g. 
spark.shuffle.memoryFraction which you also set the up limit.
 
Best Regards,
Raymond Liu
 
From: 牛兆捷 [mailto:nzjem...@gmail.com] 
Sent: Thursday, September 04, 2014 2:27 PM
To: Patrick Wendell
Cc: u...@spark.apache.org; dev@spark.apache.org
Subject: Re: memory size for caching RDD
 
But is it possible to make t resizable? When we don't have many RDD to cache, 
we can give some memory to others.
 
2014-09-04 13:45 GMT+08:00 Patrick Wendell :
Changing this is not supported, it si immutable similar to other spark
configuration settings.

On Wed, Sep 3, 2014 at 8:13 PM, 牛兆捷  wrote:
> Dear all:
>
> Spark uses memory to cache RDD and the memory size is specified by
> "spark.storage.memoryFraction".
>
> One the Executor starts, does Spark support adjusting/resizing memory size
> of this part dynamically?
>
> Thanks.
>
> --
> *Regards,*
> *Zhaojie*



-- 
Regards,
Zhaojie
 



-- 
Regards,
Zhaojie