Re: Accessing Web UI

Try http://localhost:4040 On Mon, Feb 22, 2016 at 8:23 AM, Vasanth Bhat wrote: > Thanks Gourav, Eduardo > > I tried http://localhost:8080 and http://OAhtvJ5MCA:8080/ . Both > cases the forefox just hangs. > > Also I tried with lynx text based browser. I get the

Re: spark-submit stuck and no output in console

Sonal, SparkPi couldn't run as well. Stuck to the screen with no output hadoop-user@yks-hadoop-m01:/usr/local/spark$ ./bin/run-example SparkPi On Tue, Nov 17, 2015 at 12:22 PM, Steve Loughran wrote: > 48 hours is one of those kerberos warning times (as is 24h, 72h and 7

Re: spark-submit stuck and no output in console

Our hadoop NFS Gateway seems to be malfunctioning. I basically restart it. Now spark jobs have resumed successfully. Problem solved.

Re: spark-submit stuck and no output in console

Anyone experienced this issue as well? On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi <drey...@gmail.com> wrote: > > Or are you saying that the Java process never even starts? > > > Exactly. > > Here's what I got back from jstack as expected: > > hadoop-user@yk

Re: spark-submit stuck and no output in console

conf.com/big-data-conference-sg-2015/public/schedule/detail/44606> > Reifier at Spark Summit 2015 > <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/> > > <http://in.linkedin.com/in/sonalgoyal> > > > > On Tue, Nov

Re: spark-submit stuck and no output in console

"$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@" On Mon, Nov 16, 2015 at 5:22 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Which release of Spark are you using ? > > Can you take stack trace and pastebin it ? > > Thanks > &

spark-submit stuck and no output in console

./spark-submit --class com.migration.UpdateProfiles --executor-memory 8g ~/migration-profiles-0.1-SNAPSHOT.jar is stuck and outputs nothing to the console. What could be the cause of this? Current max heap size is 1.75g and it's only using 1g.

Re: spark-submit stuck and no output in console

> Or are you saying that the Java process never even starts? Exactly. Here's what I got back from jstack as expected: hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 31316: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be

Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher

Thank you. That seems to resolve it. On Fri, Nov 6, 2015 at 11:46 PM, Ted Yu <yuzhih...@gmail.com> wrote: > You mentioned resourcemanager but not nodemanagers. > > I think you need to install Spark on nodes running nodemanagers. > > Cheers > > On Fri, Nov 6, 2015 at 1:

Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher

Hi, I have a YARN hadoop setup of 8 nodes (7 datanodes, 1 namenode and resourcemaneger). I have Spark setup only on the namenode/resource manager. Do I need to have Spark installed on the datanodes? I asked because I'm getting below error when I run a Spark job through spark-submit: Error:

Futures timed out after [120 seconds].

Hi, I'm running a Spark standalone in cluster mode (1 master, 2 workers). Everything has failed including spark-submit with errors such as "Caused by: java.lang.ClassNotFoundException: com.migration.App$$anonfun$upsert$1" Now, I've reverted back to submitting jobs through scala apps. Any ideas

Re: Executor app-20151104202102-0000 finished with state EXITED

) On Wed, Nov 4, 2015 at 9:28 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Something like this: > conf.setMaster("local[3]") > > On Wed, Nov 4, 2015 at 11:08 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > >> Thanks Ted. >> >> Where would y

Re: Executor app-20151104202102-0000 finished with state EXITED

;true") conf.set("spark.executor.memory", "5g") On Wed, Nov 4, 2015 at 9:04 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Have you tried using -Dspark.master=local ? > > Cheers > > On Wed, Nov 4, 2015 at 10:47 AM, Kayode Odeyemi <drey...@gmail.com> wro

SPARK_SSH_FOREGROUND format

From http://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts : If you do not have a password-less setup, you can set the environment > variable SPARK_SSH_FOREGROUND and serially provide a password for each > worker. > What does "serially provide a password for each

Executor app-20151104202102-0000 finished with state EXITED

Hi, I can't seem to understand why all created executors always fail. I have a Spark standalone cluster setup make up of 2 workers and 1 master. My spark-env looks like this: SPARK_MASTER_IP=192.168.2.11 SPARK_LOCAL_IP=192.168.2.11 SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=4"

Re: Maven build failed (Spark master)

r command. > > On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <drey...@gmail.com> > wrote: > >> I used this command which is synonymous to what you have: >> >> ./make-distribution.sh --name spark-latest --tgz --mvn mvn >> -Dhadoop.version=2.6.0 -Phadoop-

Re: Maven build failed (Spark master)

e custom-spark --tgz -Phadoop-2.4 -Phive >> -Phive-thriftserver -Pyarn >> >> spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz was generated (with patch from >> SPARK-11348) >> >> Can you try above command ? >> >> Thanks >> >> On Tue, Oct 27, 20

Re: Maven build failed (Spark master)

Seems the build and directory structure in dist is similar to the .gz file downloaded from the downloads page. Can the dist directory be used as is? On Tue, Oct 27, 2015 at 4:03 PM, Kayode Odeyemi <drey...@gmail.com> wrote: > Ted, I switched to this: > > ./make-distribution.

Re: Maven build failed (Spark master)

s > > On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > > Thank you. > > But I'm getting same warnings and it's still preventing the archive from > being generated. > > I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file >

Re: Maven build failed (Spark master)

t/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-2.gz.parquet > > ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-5.gz.parquet > > On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <drey...@gmail.com> > wrote:

Re: Maven build failed (Spark master)

directory > (same directory that make_distribution is in) > > > > On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > >> Hi, >> >> The ./make_distribution task completed. However, I can't seem to locate >> the >> .tar.g

Loading binary files from NFS share

Hi, Is it possible to load binary files from NFS share like this: sc.binaryFiles("nfs://host/mountpath") I understand that it takes a path, but want to know if it allows protocol. Appreciate your help.

Re: Maven build failed (Spark master)

Hi, The ./make_distribution task completed. However, I can't seem to locate the .tar.gz file. Where does Spark save this? or should I just work with the dist directory? On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <drey...@gmail.com> wrote: > I saw this when I tested manually

Maven build failed (Spark master)

Hi, I can't seem to get a successful maven build. Please see command output below: bash-3.2$ ./make-distribution.sh --name spark-latest --tgz --mvn mvn -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean package +++ dirname ./make-distribution.sh ++ cd . ++ pwd +

Re: Maven build failed (Spark master)

ual error output from Maven. I have a strong > guess that you haven't set MAVEN_OPTS to increase the memory Maven can > use. > > On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > > Hi, > > > > I can't seem to get a successful mav

Re: sqlContext load by offset

When I use that I get a "Caused by: org.postgresql.util.PSQLException: ERROR: column "none" does not exist" On Thu, Oct 22, 2015 at 9:31 PM, Kayode Odeyemi <drey...@gmail.com> wrote: > Hi, > > I've trying to load a postgres table using the following expressi

Fwd: sqlContext load by offset

Hi, I've trying to load a postgres table using the following expression: val cachedIndex = cache.get("latest_legacy_group_index") val mappingsDF = sqlContext.load("jdbc", Map( "url" -> Config.dataSourceUrl(mode, Some("mappings")), "dbtable" -> s"(select userid, yid, username from legacyusers