[jira] [Commented] (SPARK-2026) Maven hadoop* Profiles Should Set the expected Hadoop Version.
[
https://issues.apache.org/jira/browse/SPARK-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020080#comment-14020080
]
Bernardo Gomez Palacio commented on SPARK-2026:
---
https://github.com/apache/spark/pull/998
Maven hadoop* Profiles Should Set the expected Hadoop Version.
Key: SPARK-2026
URL: https://issues.apache.org/jira/browse/SPARK-2026
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 1.0.0
Reporter: Bernardo Gomez Palacio
The Maven Profiles that refer to _hadoopX_, e.g. hadoop2.4, should set the
expected _hadoop.version_.
e.g.
{code}
profile
idhadoop-2.4/id
properties
protobuf.version2.5.0/protobuf.version
jets3t.version0.9.0/jets3t.version
/properties
/profile
{code}
as it is suggested
{code}
profile
idhadoop-2.4/id
properties
hadoop.version2.4.0/hadoop.version
yarn.version${hadoop.version}/yarn.version
protobuf.version2.5.0/protobuf.version
jets3t.version0.9.0/jets3t.version
/properties
/profile
{code}
Builds can still define the -Dhadoop.version option but this will correctly
default the Hadoop Version to the one that is expected according the profile
that is selected.
e.g.
{code}
$ mvn -P hadoop-2.4,yarn clean compile
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1841) update scalatest to version 2.1.5
[ https://issues.apache.org/jira/browse/SPARK-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020211#comment-14020211 ] Bernardo Gomez Palacio commented on SPARK-1841: --- [~gq] thank you for addressing this! update scalatest to version 2.1.5 - Key: SPARK-1841 URL: https://issues.apache.org/jira/browse/SPARK-1841 Project: Spark Issue Type: Sub-task Components: Spark Core Reporter: Guoqiang Li Assignee: Guoqiang Li Fix For: 1.1.0 scalatest 1.9.* not support Scala 2.11 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-2026) Maven hadoop* Profiles Should Set the expected Hadoop Version.
[
https://issues.apache.org/jira/browse/SPARK-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019279#comment-14019279
]
Bernardo Gomez Palacio commented on SPARK-2026:
---
I'll submit a PR [~srowen]. I am not using Hadoop 0.23 but my guess is that
using 0.23.10 as default will suffice.
Maven hadoop* Profiles Should Set the expected Hadoop Version.
Key: SPARK-2026
URL: https://issues.apache.org/jira/browse/SPARK-2026
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 1.0.0
Reporter: Bernardo Gomez Palacio
The Maven Profiles that refer to _hadoopX_, e.g. hadoop2.4, should set the
expected _hadoop.version_.
e.g.
{code}
profile
idhadoop-2.4/id
properties
protobuf.version2.5.0/protobuf.version
jets3t.version0.9.0/jets3t.version
/properties
/profile
{code}
as it is suggested
{code}
profile
idhadoop-2.4/id
properties
hadoop.version2.4.0/hadoop.version
yarn.version${hadoop.version}/yarn.version
protobuf.version2.5.0/protobuf.version
jets3t.version0.9.0/jets3t.version
/properties
/profile
{code}
Builds can still define the -Dhadoop.version option but this will correctly
default the Hadoop Version to the one that is expected according the profile
that is selected.
e.g.
{code}
$ mvn -P hadoop-2.4,yarn clean compile
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (SPARK-2026) Maven hadoop* Profiles Should Set the expected Hadoop Version.
Bernardo Gomez Palacio created SPARK-2026:
-
Summary: Maven hadoop* Profiles Should Set the expected Hadoop
Version.
Key: SPARK-2026
URL: https://issues.apache.org/jira/browse/SPARK-2026
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 1.0.0
Reporter: Bernardo Gomez Palacio
The Maven Profiles that refer to _hadoopX_, e.g. hadoop2.4, should set the
expected _hadoop.version_.
e.g.
{code}
profile
idhadoop-2.4/id
properties
protobuf.version2.5.0/protobuf.version
jets3t.version0.9.0/jets3t.version
/properties
/profile
{code}
as it is suggested
{code}
profile
idhadoop-2.4/id
properties
hadoop.version2.4.0/hadoop.version
yarn.version${hadoop.version}/yarn.version
protobuf.version2.5.0/protobuf.version
jets3t.version0.9.0/jets3t.version
/properties
/profile
{code}
Builds can still define the -Dhadoop.version option but this will correctly
default the Hadoop Version to the one that is expected according the profile
that is selected.
e.g.
{code}
$ mvn -P hadoop-2.4,yarn clean compile
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (SPARK-1433) Upgrade Mesos dependency to 0.17.0
[ https://issues.apache.org/jira/browse/SPARK-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993999#comment-13993999 ] Bernardo Gomez Palacio commented on SPARK-1433: --- I agree with [~tstclair], https://github.com/berngp/spark/commit/f8ad11b436235006acf942f33a77b6758b45d17e Upgrade Mesos dependency to 0.17.0 -- Key: SPARK-1433 URL: https://issues.apache.org/jira/browse/SPARK-1433 Project: Spark Issue Type: Task Reporter: Sandeep Singh Assignee: Sandeep Singh Priority: Minor Fix For: 1.0.0 Mesos 0.13.0 was released 6 months ago. Upgrade Mesos dependency to 0.17.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (SPARK-1433) Upgrade Mesos dependency to 0.17.0
[ https://issues.apache.org/jira/browse/SPARK-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993999#comment-13993999 ] Bernardo Gomez Palacio edited comment on SPARK-1433 at 5/10/14 12:16 AM: - I agree with [~tstclair]. * Changes to use Mesos 0.18.1 with shaded protobufs can be found [here|https://github.com/berngp/spark/tree/feature/SPARK-1433] was (Author: berngp): I agree with [~tstclair], https://github.com/berngp/spark/commit/f8ad11b436235006acf942f33a77b6758b45d17e Upgrade Mesos dependency to 0.17.0 -- Key: SPARK-1433 URL: https://issues.apache.org/jira/browse/SPARK-1433 Project: Spark Issue Type: Task Reporter: Sandeep Singh Assignee: Sandeep Singh Priority: Minor Fix For: 1.0.0 Mesos 0.13.0 was released 6 months ago. Upgrade Mesos dependency to 0.17.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1764) EOF reached before Python server acknowledged
[ https://issues.apache.org/jira/browse/SPARK-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995484#comment-13995484 ] Bernardo Gomez Palacio commented on SPARK-1764: --- We just ran `sc.parallelize(range(100)).map(lambda n: n * 2).collect()` on a Mesos 0.18.1 cluster with the latest spark and it worked. Could you confirm the Spark Mesos version you are using (if using master please include the sha/commit hash). EOF reached before Python server acknowledged - Key: SPARK-1764 URL: https://issues.apache.org/jira/browse/SPARK-1764 Project: Spark Issue Type: Bug Components: Mesos, PySpark Affects Versions: 1.0.0 Reporter: Bouke van der Bijl Priority: Blocker Labels: mesos, pyspark I'm getting EOF reached before Python server acknowledged while using PySpark on Mesos. The error manifests itself in multiple ways. One is: 14/05/08 18:10:40 ERROR DAGSchedulerActorSupervisor: eventProcesserActor failed due to the error EOF reached before Python server acknowledged; shutting down SparkContext And the other has a full stacktrace: 14/05/08 18:03:06 ERROR OneForOneStrategy: EOF reached before Python server acknowledged org.apache.spark.SparkException: EOF reached before Python server acknowledged at org.apache.spark.api.python.PythonAccumulatorParam.addInPlace(PythonRDD.scala:416) at org.apache.spark.api.python.PythonAccumulatorParam.addInPlace(PythonRDD.scala:387) at org.apache.spark.Accumulable.$plus$plus$eq(Accumulators.scala:71) at org.apache.spark.Accumulators$$anonfun$add$2.apply(Accumulators.scala:279) at org.apache.spark.Accumulators$$anonfun$add$2.apply(Accumulators.scala:277) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at org.apache.spark.Accumulators$.add(Accumulators.scala:277) at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:818) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1204) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) This error causes the SparkContext to shutdown. I have not been able to reliably reproduce this bug, it seems to happen randomly, but if you run enough tasks on a SparkContext it'll hapen eventually -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SPARK-1806) Upgrade to Mesos 0.18.1 with Shaded Protobuf
Bernardo Gomez Palacio created SPARK-1806: - Summary: Upgrade to Mesos 0.18.1 with Shaded Protobuf Key: SPARK-1806 URL: https://issues.apache.org/jira/browse/SPARK-1806 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.0.0, 1.1.0, 1.0.1 Reporter: Bernardo Gomez Palacio Upgrade Spark to depend on Mesos 0.18.1 with shaded protobuf. This version of Mesos does not externalize its dependency on the protobuf version (now shaded through the namespace org.apache.mesos.protobuf) and therefore facilitates integration with systems that do depend on specific versions of protobufs such as Hadoop 1.0.x, 2.x, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1806) Upgrade to Mesos 0.18.1 with Shaded Protobuf
[ https://issues.apache.org/jira/browse/SPARK-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995265#comment-13995265 ] Bernardo Gomez Palacio commented on SPARK-1806: --- Should close SPARK-1433 Upgrade to Mesos 0.18.1 with Shaded Protobuf Key: SPARK-1806 URL: https://issues.apache.org/jira/browse/SPARK-1806 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.0.0, 1.1.0, 1.0.1 Reporter: Bernardo Gomez Palacio Labels: mesos Upgrade Spark to depend on Mesos 0.18.1 with shaded protobuf. This version of Mesos does not externalize its dependency on the protobuf version (now shaded through the namespace org.apache.mesos.protobuf) and therefore facilitates integration with systems that do depend on specific versions of protobufs such as Hadoop 1.0.x, 2.x, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1806) Upgrade to Mesos 0.18.1 with Shaded Protobuf
[ https://issues.apache.org/jira/browse/SPARK-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995382#comment-13995382 ] Bernardo Gomez Palacio commented on SPARK-1806: --- Thanks [~pwendell] for addressing this so quickly! Upgrade to Mesos 0.18.1 with Shaded Protobuf Key: SPARK-1806 URL: https://issues.apache.org/jira/browse/SPARK-1806 Project: Spark Issue Type: Dependency upgrade Components: Spark Core Affects Versions: 1.0.0, 1.1.0, 1.0.1 Reporter: Bernardo Gomez Palacio Labels: mesos Fix For: 1.0.0 Upgrade Spark to depend on Mesos 0.18.1 with shaded protobuf. This version of Mesos does not externalize its dependency on the protobuf version (now shaded through the namespace org.apache.mesos.protobuf) and therefore facilitates integration with systems that do depend on specific versions of protobufs such as Hadoop 1.0.x, 2.x, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Closed] (SPARK-1186) Enrich the Spark Shell to support additional arguments.
[
https://issues.apache.org/jira/browse/SPARK-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bernardo Gomez Palacio closed SPARK-1186.
-
Resolution: Fixed
Fix Version/s: 1.0.0
Enrich the Spark Shell to support additional arguments.
---
Key: SPARK-1186
URL: https://issues.apache.org/jira/browse/SPARK-1186
Project: Spark
Issue Type: Improvement
Affects Versions: 0.9.0
Reporter: Bernardo Gomez Palacio
Fix For: 1.0.0
Enrich the Spark Shell functionality to support the following options.
{code:title=spark-shell.sh|borderStyle=solid}
Usage: spark-shell [OPTIONS]
OPTIONS:
-h --help : Print this help information.
-c --cores : The maximum number of cores to be
used by the Spark Shell.
-em --executor-memory: The memory used by each executor of the Spark
Shell, the number
is followed by m for
megabytes or g for gigabytes, e.g. 1g.
-dm --driver-memory : The memory used by the Spark Shell, the
number is followed
by m for megabytes or g for
gigabytes, e.g. 1g.
-m --master : A full string that describes the
Spark Master, defaults to local
e.g.
spark://localhost:7077.
--log-conf : Enables logging of the supplied
SparkConf as INFO at start of the
Spark Context.
e.g.
spark-shell -m spark://localhost:7077 -c 4 -dm 512m -em 2g
{code}
**Note**: the options described above are not visually aligned due JIRA's
rendering, in the bash CLI they are.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (SPARK-1522) YARN ClientBase will throw a NPE if there is no YARN application specific classpath.
Bernardo Gomez Palacio created SPARK-1522: - Summary: YARN ClientBase will throw a NPE if there is no YARN application specific classpath. Key: SPARK-1522 URL: https://issues.apache.org/jira/browse/SPARK-1522 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 0.9.0, 1.0.0, 0.9.1 Reporter: Bernardo Gomez Palacio Priority: Critical The current implementation of ClientBase.getDefaultYarnApplicationClasspath inspects the MRJobConfig class for the field DEFAULT_YARN_APPLICATION_CLASSPATH when it should be really looking into YarnConfiguration. If the Application Configuration has no yarn.application.classpath defined a NPE exception will be thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1522) YARN ClientBase will throw a NPE if there is no YARN application specific classpath.
[ https://issues.apache.org/jira/browse/SPARK-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972803#comment-13972803 ] Bernardo Gomez Palacio commented on SPARK-1522: --- https://github.com/apache/spark/pull/433 YARN ClientBase will throw a NPE if there is no YARN application specific classpath. Key: SPARK-1522 URL: https://issues.apache.org/jira/browse/SPARK-1522 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 0.9.0, 1.0.0, 0.9.1 Reporter: Bernardo Gomez Palacio Priority: Critical Labels: YARN The current implementation of ClientBase.getDefaultYarnApplicationClasspath inspects the MRJobConfig class for the field DEFAULT_YARN_APPLICATION_CLASSPATH when it should be really looking into YarnConfiguration. If the Application Configuration has no yarn.application.classpath defined a NPE exception will be thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
