Re: Add hot-deploy capability in Spark Shell

[Kai Chen]

I don't.  The hot-deploy shouldn't happen while there is a job running.  At
least in the REPL it won't make much sense.  It's a development-only
feature to shorten the iterative coding cycle.  In production environment,
this is not enabled ... though there might be situations where it would be
desirable.  But currently I'm not handling that, as it's much more complex.

On Mon, Jun 6, 2016 at 4:16 PM, Reynold Xin  wrote:

> Thanks for the email. How do you deal with in-memory state that reference
> the classes? This can happen in both streaming and caching in RDD and
> temporary view creation in SQL.
>
> On Mon, Jun 6, 2016 at 3:40 PM, S. Kai Chen 
> wrote:
>
>> Hi,
>>
>> We use spark-shell heavily for ad-hoc data analysis as well as iterative
>> development of the analytics code. A common workflow consists the following
>> steps:
>>
>>1. Write a small Scala module, assemble the fat jar
>>2. Start spark-shell with the assembly jar file
>>3. Try out some ideas in the shell, then capture the code back into
>>the module
>>4. Go back to step 1 and restart the shell
>>
>> This is very similar to what people do in web-app development. And the
>> pain point is similar: in web-app development, a lot of time is spent
>> waiting for new code to be deployed; here, a lot of time is spent waiting
>> for Spark to restart. Having the ability to hot-deploy code in the REPL
>> would help a lot, just as being able to hot-deploy in containers like Play,
>> or using JRebel, has helped boost productivity tremendously.
>>
>> I do have code that works with the 1.5.2 release.  Is this something
>> that's interesting enough to be included in Spark proper?  If so, should I
>> create a Jira ticket or github PR for the master branch?
>>
>>
>> Cheers,
>>
>> Kai
>>
>
>

Re: Add hot-deploy capability in Spark Shell

[Reynold Xin]

Thanks for the email. How do you deal with in-memory state that reference
the classes? This can happen in both streaming and caching in RDD and
temporary view creation in SQL.

On Mon, Jun 6, 2016 at 3:40 PM, S. Kai Chen  wrote:

> Hi,
>
> We use spark-shell heavily for ad-hoc data analysis as well as iterative
> development of the analytics code. A common workflow consists the following
> steps:
>
>1. Write a small Scala module, assemble the fat jar
>2. Start spark-shell with the assembly jar file
>3. Try out some ideas in the shell, then capture the code back into
>the module
>4. Go back to step 1 and restart the shell
>
> This is very similar to what people do in web-app development. And the
> pain point is similar: in web-app development, a lot of time is spent
> waiting for new code to be deployed; here, a lot of time is spent waiting
> for Spark to restart. Having the ability to hot-deploy code in the REPL
> would help a lot, just as being able to hot-deploy in containers like Play,
> or using JRebel, has helped boost productivity tremendously.
>
> I do have code that works with the 1.5.2 release.  Is this something
> that's interesting enough to be included in Spark proper?  If so, should I
> create a Jira ticket or github PR for the master branch?
>
>
> Cheers,
>
> Kai
>

Add hot-deploy capability in Spark Shell

[S. Kai Chen]

Hi,

We use spark-shell heavily for ad-hoc data analysis as well as iterative
development of the analytics code. A common workflow consists the following
steps:

   1. Write a small Scala module, assemble the fat jar
   2. Start spark-shell with the assembly jar file
   3. Try out some ideas in the shell, then capture the code back into the
   module
   4. Go back to step 1 and restart the shell

This is very similar to what people do in web-app development. And the pain
point is similar: in web-app development, a lot of time is spent waiting
for new code to be deployed; here, a lot of time is spent waiting for Spark
to restart. Having the ability to hot-deploy code in the REPL would help a
lot, just as being able to hot-deploy in containers like Play, or using
JRebel, has helped boost productivity tremendously.

I do have code that works with the 1.5.2 release.  Is this something that's
interesting enough to be included in Spark proper?  If so, should I create
a Jira ticket or github PR for the master branch?


Cheers,

Kai

Re: apologies for flaky BlacklistIntegrationSuite

[Reynold Xin]

Thanks for fixing it!


On Mon, Jun 6, 2016 at 1:49 PM, Imran Rashid  wrote:

> Hi all,
>
> just a heads up, I introduced a flaky test, BlacklistIntegrationSuite, a
> week ago or so.  I *thought* I had solved the problems, but turns out there
> was more flakiness remaining.  for now I've just turned the tests off, so
> if you this has led to failures for you, just re-trigger your build.
>
> Opened a jira to fix it for real here if you're interested:
> https://issues.apache.org/jira/browse/SPARK-15783
>
> Again apologies for the flakiness in the build.
>
> Imran
>

Re: Mesos and No transport is loaded for protocol

[Timothy Chen]

Hi,

How did you package the spark.tgz, and are you running the same code that you 
packaged when you ran spark submit?

And what is your settings for spark look like?

Tim


> On Jun 6, 2016, at 12:13 PM, thibaut  wrote:
> 
> Hi there,
> 
> I an trying to configure Spark for running on top of Mesos. But every time I 
> send a job, it fails. I can see mesos downloading correctly the spark.tgz but 
> I have this errors at the end :
> 
> 
> Any idea ? I did not find anything for solving my issue..  Is it my cluster ? 
> Spark ? both ? Thanking you in advance.
> Thibaut
> 
> I0606 15:06:35.628329 16520 fetcher.cpp:456] Fetched 
> 'http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' to 
> '/tmp/mesos/slaves/c58064f7-88b6-438d-b76f-fc28c6cc51a1-S0/frameworks/c58064f7-88b6-438d-b76f-fc28c6cc51a1-0079/executors/3/runs/23913146-d87f-445c-9f6b-f412ad2cbbd7/spark-1.5.1-bin-hadoop2.6.tgz'
> I0606 15:06:35.687414 16527 exec.cpp:143] Version: 0.28.1
> I0606 15:06:35.691270 16540 exec.cpp:217] Executor registered on slave 
> c58064f7-88b6-438d-b76f-fc28c6cc51a1-S0
> Using Spark's default log4j profile: 
> org/apache/spark/log4j-defaults.properties
> 16/06/06 15:06:36 INFO CoarseGrainedExecutorBackend: Registered signal 
> handlers for [TERM, HUP, INT]
> 16/06/06 15:06:36 WARN NativeCodeLoader: Unable to load native-hadoop library 
> for your platform... using builtin-java classes where applicable
> 16/06/06 15:06:37 INFO SecurityManager: Changing view acls to: thibautg
> 16/06/06 15:06:37 INFO SecurityManager: Changing modify acls to: thibautg
> 16/06/06 15:06:37 INFO SecurityManager: SecurityManager: authentication 
> disabled; ui acls disabled; users with view permissions: Set(thibautg); users 
> with modify permissions: Set(thibautg)
> 16/06/06 15:06:37 INFO Slf4jLogger: Slf4jLogger started
> 16/06/06 15:06:38 INFO Remoting: Starting remoting
> 16/06/06 15:06:38 INFO Remoting: Remoting started; listening on addresses 
> :[akka.tcp://driverPropsFetcher@141.213.4.119:56419]
> 16/06/06 15:06:38 INFO Utils: Successfully started service 
> 'driverPropsFetcher' on port 56419.
> Exception in thread "main" akka.remote.RemoteTransportException: No transport 
> is loaded for protocol: [spark], available protocols: [akka.tcp]
>   at akka.remote.Remoting$.localAddressForRemote(Remoting.scala:87)
>   at akka.remote.Remoting.localAddressForRemote(Remoting.scala:129)
>   at 
> akka.remote.RemoteActorRefProvider.rootGuardianAt(RemoteActorRefProvider.scala:338)
>   at 
> akka.actor.ActorRefFactory$class.actorSelection(ActorRefProvider.scala:318)
>   at akka.actor.ActorSystem.actorSelection(ActorSystem.scala:272)
>   at 
> org.apache.spark.rpc.akka.AkkaRpcEnv.asyncSetupEndpointRefByURI(AkkaRpcEnv.scala:216)
>   at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:98)
>   at 
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:162)
>   at 
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
>   at 
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
>   at 
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:149)
>   at 
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:250)
>   at 
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> 
>  

apologies for flaky BlacklistIntegrationSuite

[Imran Rashid]

Hi all,

just a heads up, I introduced a flaky test, BlacklistIntegrationSuite, a
week ago or so.  I *thought* I had solved the problems, but turns out there
was more flakiness remaining.  for now I've just turned the tests off, so
if you this has led to failures for you, just re-trigger your build.

Opened a jira to fix it for real here if you're interested:
https://issues.apache.org/jira/browse/SPARK-15783

Again apologies for the flakiness in the build.

Imran

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Imran Rashid]

I've been a bit on the fence on this, but I agree that Luciano makes a
compelling reason for why we really should publish things to maven
central.  Sure we slightly increase the risk somebody refers to the preview
release too late, but really that is their own fault.

And I also I agree with comments from Sean and Mark that this is *not* a
"Databricks vs The World" scenario at all.

On Mon, Jun 6, 2016 at 2:13 PM, Luciano Resende 
wrote:

>
>
> On Mon, Jun 6, 2016 at 12:05 PM, Reynold Xin  wrote:
>
>> The bahir one was a good argument actually. I just clicked the button to
>> push it into Maven central.
>>
>>
> Thank You !!!
>
>

Mesos and No transport is loaded for protocol

[thibaut]

Hi there,

I an trying to configure Spark for running on top of Mesos. But every time I 
send a job, it fails. I can see mesos downloading correctly the spark.tgz but I 
have this errors at the end :


Any idea ? I did not find anything for solving my issue..  Is it my cluster ? 
Spark ? both ? Thanking you in advance.
Thibaut

I0606 15:06:35.628329 16520 fetcher.cpp:456] Fetched 
'http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' to 
'/tmp/mesos/slaves/c58064f7-88b6-438d-b76f-fc28c6cc51a1-S0/frameworks/c58064f7-88b6-438d-b76f-fc28c6cc51a1-0079/executors/3/runs/23913146-d87f-445c-9f6b-f412ad2cbbd7/spark-1.5.1-bin-hadoop2.6.tgz'
I0606 15:06:35.687414 16527 exec.cpp:143] Version: 0.28.1
I0606 15:06:35.691270 16540 exec.cpp:217] Executor registered on slave 
c58064f7-88b6-438d-b76f-fc28c6cc51a1-S0
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/06/06 15:06:36 INFO CoarseGrainedExecutorBackend: Registered signal handlers 
for [TERM, HUP, INT]
16/06/06 15:06:36 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
16/06/06 15:06:37 INFO SecurityManager: Changing view acls to: thibautg
16/06/06 15:06:37 INFO SecurityManager: Changing modify acls to: thibautg
16/06/06 15:06:37 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users with view permissions: Set(thibautg); users 
with modify permissions: Set(thibautg)
16/06/06 15:06:37 INFO Slf4jLogger: Slf4jLogger started
16/06/06 15:06:38 INFO Remoting: Starting remoting
16/06/06 15:06:38 INFO Remoting: Remoting started; listening on addresses 
:[akka.tcp://driverPropsFetcher@141.213.4.119:56419]
16/06/06 15:06:38 INFO Utils: Successfully started service 'driverPropsFetcher' 
on port 56419.
Exception in thread "main" akka.remote.RemoteTransportException: No transport 
is loaded for protocol: [spark], available protocols: [akka.tcp]
at akka.remote.Remoting$.localAddressForRemote(Remoting.scala:87)
at akka.remote.Remoting.localAddressForRemote(Remoting.scala:129)
at 
akka.remote.RemoteActorRefProvider.rootGuardianAt(RemoteActorRefProvider.scala:338)
at 
akka.actor.ActorRefFactory$class.actorSelection(ActorRefProvider.scala:318)
at akka.actor.ActorSystem.actorSelection(ActorSystem.scala:272)
at 
org.apache.spark.rpc.akka.AkkaRpcEnv.asyncSetupEndpointRefByURI(AkkaRpcEnv.scala:216)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:98)
at 
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:162)
at 
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
at 
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
at 
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:149)
at 
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:250)
at 
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

 

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Luciano Resende]

On Mon, Jun 6, 2016 at 12:05 PM, Reynold Xin  wrote:

> The bahir one was a good argument actually. I just clicked the button to
> push it into Maven central.
>
>
Thank You !!!

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Reynold Xin]

The bahir one was a good argument actually. I just clicked the button to
push it into Maven central.


On Mon, Jun 6, 2016 at 12:00 PM, Mark Hamstra 
wrote:

> Fine.  I don't feel strongly enough about it to continue to argue against
> putting the artifacts on Maven Central.
>
> On Mon, Jun 6, 2016 at 11:48 AM, Sean Owen  wrote:
>
>> Artifacts can't be removed from Maven in any normal circumstance, but,
>> it's no problem.
>>
>> The argument that people might keep using it goes for any older
>> release. Why would anyone use 1.6.0 when 1.6.1 exists? yet we keep
>> 1.6.0 just for the record and to not break builds. It may be that
>> Foobar 3.0-beta depends on 2.0.0-preview and 3.0 will shortly depend
>> on 2.0.0, but, killing the -preview artifact breaks that other
>> historical release/branch.
>>
>> I agree that "-alpha-1" would have been better. But we're talking
>> about working around pretty bone-headed behavior, to not notice what
>> version of Spark they build against, or not understand what
>> 2.0.0-preview vs 2.0.0 means in a world of semver.
>>
>> BTW Maven sorts 2.0.0-preview before 2.0.0, so 2.0.0 would show up as
>> the latest, when released, in tools like mvn
>> versions:display-dependency-updates. You could exclude the preview
>> release by requiring version [2.0.0,).
>>
>> On Mon, Jun 6, 2016 at 7:19 PM, Mark Hamstra 
>> wrote:
>> > Precisely because the naming of the preview artifacts has to fall
>> outside of
>> > the normal versioning, I can easily see incautious Maven users a few
>> months
>> > from now mistaking the preview artifacts as spark-2.0-something-special
>> > instead of spark-2.0-something-stale.
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Sean Owen]

Artifacts can't be removed from Maven in any normal circumstance, but,
it's no problem.

The argument that people might keep using it goes for any older
release. Why would anyone use 1.6.0 when 1.6.1 exists? yet we keep
1.6.0 just for the record and to not break builds. It may be that
Foobar 3.0-beta depends on 2.0.0-preview and 3.0 will shortly depend
on 2.0.0, but, killing the -preview artifact breaks that other
historical release/branch.

I agree that "-alpha-1" would have been better. But we're talking
about working around pretty bone-headed behavior, to not notice what
version of Spark they build against, or not understand what
2.0.0-preview vs 2.0.0 means in a world of semver.

BTW Maven sorts 2.0.0-preview before 2.0.0, so 2.0.0 would show up as
the latest, when released, in tools like mvn
versions:display-dependency-updates. You could exclude the preview
release by requiring version [2.0.0,).

On Mon, Jun 6, 2016 at 7:19 PM, Mark Hamstra  wrote:
> Precisely because the naming of the preview artifacts has to fall outside of
> the normal versioning, I can easily see incautious Maven users a few months
> from now mistaking the preview artifacts as spark-2.0-something-special
> instead of spark-2.0-something-stale.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Ovidiu-Cristian MARCU]

+1 for moving this discussion to a proactive new (alpha/beta) release of Apache 
Spark 2.0!

> On 06 Jun 2016, at 20:25, Ovidiu Cristian Marcu  wrote:
> 
> Any chance to start preparing a new alpha/beta release for 2.0 this month or 
> the preview will be pushed to maven and considered an alpha?
> 
> Sent from TypeApp 
> 
> On Jun 6, 2016, at 20:12, Matei Zaharia  > wrote:
> Is there any way to remove artifacts from Maven Central? Maybe that would 
> help clean these things up long-term, though it would create problems for 
> users who for some reason decide to rely on these previews. 
> 
> In any case, if people are *really* concerned about this, we should just put 
> it there. My thought was that it's better for users to do something special 
> to link to this release (e.g. add a reference to the staging repo) so that 
> they are more likely to know that it's a special, unstable thing. Same thing 
> they do to use snapshots. 
> 
> Matei
> 
> On Mon, Jun 6, 2016 at 10:49 AM, Luciano Resende  > wrote: 
> 
> 
> On Mon, Jun 6, 2016 at 10:08 AM, Mark Hamstra  > wrote:
> I still don't know where this "severely compromised builds of limited 
> usefulness" thing comes from? what's so bad? You didn't veto its 
> release, after all.
> 
> I simply mean that it was released with the knowledge that there are still 
> significant bugs in the preview that definitely would warrant a veto if this 
> were intended to be on a par with other releases.  There have been repeated 
> announcements to that effect, but developers finding the preview artifacts on 
> Maven Central months from now may well not also see those announcements and 
> related discussion.  The artifacts will be very stale and no longer useful 
> for their limited testing purpose, but will persist in the repository.  
> 
> 
> A few months from now, why would a developer choose a preview, alpha, beta 
> compared to the GA 2.0 release ? 
> 
> As for the being stale part, this is true for every release anyone put out 
> there. 
> 
> 
> -- 
> Luciano Resende 
> http://twitter.com/lresende1975  
> http://lresende.blogspot.com/ 

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Luciano Resende]

On Mon, Jun 6, 2016 at 11:12 AM, Matei Zaharia 
wrote:

> Is there any way to remove artifacts from Maven Central? Maybe that would
> help clean these things up long-term, though it would create problems for
> users who for some reason decide to rely on these previews.
>
> In any case, if people are *really* concerned about this, we should just
> put it there. My thought was that it's better for users to do something
> special to link to this release (e.g. add a reference to the staging repo)
> so that they are more likely to know that it's a special, unstable thing.
> Same thing they do to use snapshots.
>
> Matei
>
>
So, consider this thread started on another project :
https://www.mail-archive.com/dev@bahir.apache.org/msg00038.html

What would be your recommendation ?
   - Start a release based on Apache Spark 2.0.0 preview staging repo ? I
would  reject that...
   - Start a release on a set of artifacts that are going to be deleted ? I
would also reject that

To me, if companies are using the release on their products, and other
projects are relying on the release to provide a way for users to test,
this should be considered as any other release, published permanently,
which at some point will become obsolete and users will move on to more
stable releases.

Thanks



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Matei Zaharia]

Is there any way to remove artifacts from Maven Central? Maybe that would
help clean these things up long-term, though it would create problems for
users who for some reason decide to rely on these previews.

In any case, if people are *really* concerned about this, we should just
put it there. My thought was that it's better for users to do something
special to link to this release (e.g. add a reference to the staging repo)
so that they are more likely to know that it's a special, unstable thing.
Same thing they do to use snapshots.

Matei

On Mon, Jun 6, 2016 at 10:49 AM, Luciano Resende 
wrote:

>
>
> On Mon, Jun 6, 2016 at 10:08 AM, Mark Hamstra 
> wrote:
>
>> I still don't know where this "severely compromised builds of limited
>>> usefulness" thing comes from? what's so bad? You didn't veto its
>>> release, after all.
>>
>>
>> I simply mean that it was released with the knowledge that there are
>> still significant bugs in the preview that definitely would warrant a veto
>> if this were intended to be on a par with other releases.  There have been
>> repeated announcements to that effect, but developers finding the preview
>> artifacts on Maven Central months from now may well not also see those
>> announcements and related discussion.  The artifacts will be very stale and
>> no longer useful for their limited testing purpose, but will persist in the
>> repository.
>>
>>
> A few months from now, why would a developer choose a preview, alpha, beta
> compared to the GA 2.0 release ?
>
> As for the being stale part, this is true for every release anyone put out
> there.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Luciano Resende]

On Mon, Jun 6, 2016 at 10:08 AM, Mark Hamstra 
wrote:

> I still don't know where this "severely compromised builds of limited
>> usefulness" thing comes from? what's so bad? You didn't veto its
>> release, after all.
>
>
> I simply mean that it was released with the knowledge that there are still
> significant bugs in the preview that definitely would warrant a veto if
> this were intended to be on a par with other releases.  There have been
> repeated announcements to that effect, but developers finding the preview
> artifacts on Maven Central months from now may well not also see those
> announcements and related discussion.  The artifacts will be very stale and
> no longer useful for their limited testing purpose, but will persist in the
> repository.
>
>
A few months from now, why would a developer choose a preview, alpha, beta
compared to the GA 2.0 release ?

As for the being stale part, this is true for every release anyone put out
there.


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Luciano Resende]

On Mon, Jun 6, 2016 at 9:51 AM, Sean Owen  wrote:

> I still don't know where this "severely compromised builds of limited
> usefulness" thing comes from? what's so bad? You didn't veto its
> release, after all. And rightly so: a release doesn't mean "definitely
> works"; it means it was created the right way. It's OK to say it's
> buggy alpha software; this isn't an argument to not really release it.
>
> But aside from that: if it should be used by someone, then who did you
> have in mind?
>
> It would be coherent at least to decide not to make alpha-like
> release, but, we agreed to, which is why this argument sort of
> surprises me.
>
> I share some concerns about piling on Databricks. Nothing here is by
> nature about an organization. However, this release really began in
> response to a thread (which not everyone here can see) about
> Databricks releasing a "2.0.0 preview" option in their product before
> it existed. I presume employees of that company sort of endorse this,
> which has put this same release into the hands of not just developers
> or admins but end users -- even with caveats and warnings.
>
> (And I think that's right!)
>
>

In this case, I would only expect the 2.0.0 preview to be treated as just
any other release, period.


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Can't compile 2.0-preview with scala 2.10

[Ted Yu]

See the following from
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.10/1642/consoleFull
:

+ SBT_FLAGS+=('-Dscala-2.10')
+ ./dev/change-scala-version.sh 2.10


FYI


On Mon, Jun 6, 2016 at 10:35 AM, Franklyn D'souza <
franklyn.dso...@shopify.com> wrote:

> Hi,
>
> I've checked out the 2.0-preview and attempted to build it
> with ./dev/make-distribution.sh -Pscala-2.10
>
> However i keep getting
>
> [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @
> spark-parent_2.11 ---
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
> failed with message:
> Found Banned Dependency: org.scala-lang.modules:scala-xml_2.11:jar:1.0.2
> Found Banned Dependency: org.scalatest:scalatest_2.11:jar:2.2.6
>
> Is scala 2.10 not being supported going forward ?. If so the profile
> should probably be removed from the master pom.xml
>
>
> Thanks,
>
> Franklyn
>
>
>

Can't compile 2.0-preview with scala 2.10

[Franklyn D'souza]

Hi,

I've checked out the 2.0-preview and attempted to build it
with ./dev/make-distribution.sh -Pscala-2.10

However i keep getting

[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @
spark-parent_2.11 ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
failed with message:
Found Banned Dependency: org.scala-lang.modules:scala-xml_2.11:jar:1.0.2
Found Banned Dependency: org.scalatest:scalatest_2.11:jar:2.2.6

Is scala 2.10 not being supported going forward ?. If so the profile should
probably be removed from the master pom.xml


Thanks,

Franklyn

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Mark Hamstra]

>
> I still don't know where this "severely compromised builds of limited
> usefulness" thing comes from? what's so bad? You didn't veto its
> release, after all.


I simply mean that it was released with the knowledge that there are still
significant bugs in the preview that definitely would warrant a veto if
this were intended to be on a par with other releases.  There have been
repeated announcements to that effect, but developers finding the preview
artifacts on Maven Central months from now may well not also see those
announcements and related discussion.  The artifacts will be very stale and
no longer useful for their limited testing purpose, but will persist in the
repository.

On Mon, Jun 6, 2016 at 9:51 AM, Sean Owen  wrote:

> I still don't know where this "severely compromised builds of limited
> usefulness" thing comes from? what's so bad? You didn't veto its
> release, after all. And rightly so: a release doesn't mean "definitely
> works"; it means it was created the right way. It's OK to say it's
> buggy alpha software; this isn't an argument to not really release it.
>
> But aside from that: if it should be used by someone, then who did you
> have in mind?
>
> It would be coherent at least to decide not to make alpha-like
> release, but, we agreed to, which is why this argument sort of
> surprises me.
>
> I share some concerns about piling on Databricks. Nothing here is by
> nature about an organization. However, this release really began in
> response to a thread (which not everyone here can see) about
> Databricks releasing a "2.0.0 preview" option in their product before
> it existed. I presume employees of that company sort of endorse this,
> which has put this same release into the hands of not just developers
> or admins but end users -- even with caveats and warnings.
>
> (And I think that's right!)
>
> While I'd like to see your reasons before I'd agree with you Mark,
> yours is a feasible position; I'm not as sure how people who work for
> Databricks can argue at the same time however that this should be
> carefully guarded as an ASF release -- even with caveats and warnings.
>
> We don't need to assume bad faith -- I don't. The appearance alone is
> enough to act to make this consistent.
>
> But, I think the resolution is simple: it's not 'dangerous' to release
> this and I don't think people who say they think this really do. So
> just finish this release normally, and we're done. Even if you think
> there's an argument against it, weigh vs the problems above.
>
>
> On Mon, Jun 6, 2016 at 4:00 PM, Mark Hamstra 
> wrote:
> > This is not a Databricks vs. The World situation, and the fact that some
> > persist in forcing every issue into that frame is getting annoying.
> There
> > are good engineering and project-management reasons not to populate the
> > long-term, canonical repository of Maven artifacts with what are known
> to be
> > severely compromised builds of limited usefulness, particularly over
> time.
> > It is a legitimate dispute over whether these preview artifacts should be
> > deployed to Maven Central, not one that must be seen as Databricks
> seeking
> > improper advantage.
> >
>

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Sean Owen]

I still don't know where this "severely compromised builds of limited
usefulness" thing comes from? what's so bad? You didn't veto its
release, after all. And rightly so: a release doesn't mean "definitely
works"; it means it was created the right way. It's OK to say it's
buggy alpha software; this isn't an argument to not really release it.

But aside from that: if it should be used by someone, then who did you
have in mind?

It would be coherent at least to decide not to make alpha-like
release, but, we agreed to, which is why this argument sort of
surprises me.

I share some concerns about piling on Databricks. Nothing here is by
nature about an organization. However, this release really began in
response to a thread (which not everyone here can see) about
Databricks releasing a "2.0.0 preview" option in their product before
it existed. I presume employees of that company sort of endorse this,
which has put this same release into the hands of not just developers
or admins but end users -- even with caveats and warnings.

(And I think that's right!)

While I'd like to see your reasons before I'd agree with you Mark,
yours is a feasible position; I'm not as sure how people who work for
Databricks can argue at the same time however that this should be
carefully guarded as an ASF release -- even with caveats and warnings.

We don't need to assume bad faith -- I don't. The appearance alone is
enough to act to make this consistent.

But, I think the resolution is simple: it's not 'dangerous' to release
this and I don't think people who say they think this really do. So
just finish this release normally, and we're done. Even if you think
there's an argument against it, weigh vs the problems above.


On Mon, Jun 6, 2016 at 4:00 PM, Mark Hamstra  wrote:
> This is not a Databricks vs. The World situation, and the fact that some
> persist in forcing every issue into that frame is getting annoying.  There
> are good engineering and project-management reasons not to populate the
> long-term, canonical repository of Maven artifacts with what are known to be
> severely compromised builds of limited usefulness, particularly over time.
> It is a legitimate dispute over whether these preview artifacts should be
> deployed to Maven Central, not one that must be seen as Databricks seeking
> improper advantage.
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

subscribe

[Kishorkumar Patil]

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Nicholas Chammas]

+1 to what Mark said. I've been following this discussion and I don't
understand where the sudden "Databricks vs. everybody else" narrative came
from.

On Mon, Jun 6, 2016 at 11:00 AM Mark Hamstra 
wrote:

> This is not a Databricks vs. The World situation, and the fact that some
> persist in forcing every issue into that frame is getting annoying.  There
> are good engineering and project-management reasons not to populate the
> long-term, canonical repository of Maven artifacts with what are known to
> be severely compromised builds of limited usefulness, particularly over
> time.  It is a legitimate dispute over whether these preview artifacts
> should be deployed to Maven Central, not one that must be seen as
> Databricks seeking improper advantage.
>
> On Mon, Jun 6, 2016 at 5:34 AM, Shane Curcuru 
> wrote:
>
>>
>>
>> On 2016-06-04 18:42 (-0400), Sean Owen  wrote:
>> ...
>> > The question is, can you just not fully release it? I don't think so,
>> > even as a matter of process, and don't see a good reason not to.
>> >
>> > To Reynold's quote, I think that's suggesting that not all projects
>> > will release to a repo at all (e.g. OpenOffice?). I don't think it
>> > means you're free to not release some things to Maven, if that's
>> > appropriate and common for the type of project.
>> >
>> > Regarding risk, remember that the audience for Maven artifacts are
>> > developers, not admins or end users. I understand that developers can
>> > temporarily change their build to use a different resolver if they
>> > care, but, why? (and, where would someone figure this out?)
>> >
>> > Regardless: the 2.0.0-preview docs aren't published to go along with
>> > the source/binary releases. Those need be released to the project
>> > site, though probably under a different /preview/ path or something.
>> > If they are, is it weird that someone wouldn't find the release in the
>> > usual place in Maven then?
>> >
>> > Given that the driver of this was concern over wide access to
>> > 2.0.0-preview, I think it's best to err on the side openness vs some
>> > theoretical problem.
>>
>> The mere fact that there continues to be repeated pushback from PMC
>> members employed by DataBricks to such a reasonable and easy question to
>> answer and take action on for the benefit of all the project's users
>> raises red flags for me.
>>
>> Immaterial of the actual motivations of individual PMC members, this
>> still gives the *appearance* that DataBricks as an organization
>> effectively exercises a more than healthy amount of control over how the
>> project operates in simple, day-to-day manners.
>>
>> I strongly urge everyone participating in Apache Spark development to
>> read and take to heart this required policy for Apache projects:
>>
>>   http://community.apache.org/projectIndependence
>>
>> - Shane, speaking as an individual
>>
>> (If I were speaking in other roles I hold, I wouldn't be as polite)
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>

Re: Welcoming Yanbo Liang as a committer

[Gayathri Murali]

Congratulations Yanbo Liang! Well deserved.


On Sun, Jun 5, 2016 at 7:10 PM, Shixiong(Ryan) Zhu 
wrote:

> Congrats, Yanbo!
>
> On Sun, Jun 5, 2016 at 6:25 PM, Liwei Lin  wrote:
>
>> Congratulations Yanbo!
>>
>> On Mon, Jun 6, 2016 at 7:07 AM, Bryan Cutler  wrote:
>>
>>> Congratulations Yanbo!
>>> On Jun 5, 2016 4:03 AM, "Kousuke Saruta" 
>>> wrote:
>>>
 Congratulations Yanbo!


 - Kousuke

 On 2016/06/04 11:48, Matei Zaharia wrote:

> Hi all,
>
> The PMC recently voted to add Yanbo Liang as a committer. Yanbo has
> been a super active contributor in many areas of MLlib. Please join me in
> welcoming Yanbo!
>
> Matei
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

 -
 To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
 For additional commands, e-mail: dev-h...@spark.apache.org


>>
>

Re: Spark 2.0.0-preview artifacts still not available in Maven

[Mark Hamstra]

This is not a Databricks vs. The World situation, and the fact that some
persist in forcing every issue into that frame is getting annoying.  There
are good engineering and project-management reasons not to populate the
long-term, canonical repository of Maven artifacts with what are known to
be severely compromised builds of limited usefulness, particularly over
time.  It is a legitimate dispute over whether these preview artifacts
should be deployed to Maven Central, not one that must be seen as
Databricks seeking improper advantage.

On Mon, Jun 6, 2016 at 5:34 AM, Shane Curcuru  wrote:

>
>
> On 2016-06-04 18:42 (-0400), Sean Owen  wrote:
> ...
> > The question is, can you just not fully release it? I don't think so,
> > even as a matter of process, and don't see a good reason not to.
> >
> > To Reynold's quote, I think that's suggesting that not all projects
> > will release to a repo at all (e.g. OpenOffice?). I don't think it
> > means you're free to not release some things to Maven, if that's
> > appropriate and common for the type of project.
> >
> > Regarding risk, remember that the audience for Maven artifacts are
> > developers, not admins or end users. I understand that developers can
> > temporarily change their build to use a different resolver if they
> > care, but, why? (and, where would someone figure this out?)
> >
> > Regardless: the 2.0.0-preview docs aren't published to go along with
> > the source/binary releases. Those need be released to the project
> > site, though probably under a different /preview/ path or something.
> > If they are, is it weird that someone wouldn't find the release in the
> > usual place in Maven then?
> >
> > Given that the driver of this was concern over wide access to
> > 2.0.0-preview, I think it's best to err on the side openness vs some
> > theoretical problem.
>
> The mere fact that there continues to be repeated pushback from PMC
> members employed by DataBricks to such a reasonable and easy question to
> answer and take action on for the benefit of all the project's users
> raises red flags for me.
>
> Immaterial of the actual motivations of individual PMC members, this
> still gives the *appearance* that DataBricks as an organization
> effectively exercises a more than healthy amount of control over how the
> project operates in simple, day-to-day manners.
>
> I strongly urge everyone participating in Apache Spark development to
> read and take to heart this required policy for Apache projects:
>
>   http://community.apache.org/projectIndependence
>
> - Shane, speaking as an individual
>
> (If I were speaking in other roles I hold, I wouldn't be as polite)
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: https://issues.apache.org seems to be down for a while today.

[Jean-Baptiste Onofré]

Yes, Jira was down couple of hours ago, but it's back now.

Regards
JB

On 06/06/2016 12:51 PM, Prashant Sharma wrote:

Hi All,

https://issues.apache.org seems to be down for a while today. I was not
sure, who looks into this.

http://downorisitjustme.com/res.php?url=issues.apache.org


Thanks,
--Prashant



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org