Just in case, anyone knows how I can download Spark 1.2? It is not
>> showing up in Spark download page drop down
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
--
Best Regards,
Ayan Guha
All previous releases are available on the Release Archives
http://archive.apache.org/dist/spark/
On Tue, Oct 25, 2016 at 2:17 PM, ayan guha <guha.a...@gmail.com> wrote:
> Just in case, anyone knows how I can download Spark 1.2? It is not showing
> up in Spark download pa
archive.apache.org will always have all the releases:
http://archive.apache.org/dist/spark/
On Tue, Oct 25, 2016 at 1:17 PM ayan guha <guha.a...@gmail.com> wrote:
> Just in case, anyone knows how I can download Spark 1.2? It is not showing
> up in Spark download page drop down
>
Just in case, anyone knows how I can download Spark 1.2? It is not showing
up in Spark download page drop down
--
Best Regards,
Ayan Guha
from Spark 1.2 to
Spark
1.3
15/05/18 18:22:39 WARN TaskSetManager: Lost task 0.0 in stage 1.6 (TID 84,
cloud8-server): FetchFailed(BlockManagerId(1, cloud4-server, 7337),
shuffleId=0, mapId=9, reduceId=1, message=
org.apache.spark.shuffle.FetchFailedException: java.lang.RuntimeException
Hi, I'm getting this exception after shifting my code from Spark 1.2 to Spark
1.3
15/05/18 18:22:39 WARN TaskSetManager: Lost task 0.0 in stage 1.6 (TID 84,
cloud8-server): FetchFailed(BlockManagerId(1, cloud4-server, 7337),
shuffleId=0, mapId=9, reduceId=1, message
trace. Same works with
Spark 1.2 in same environment
val codec = classOf[some codec class]
val a = sc.textFile(/some_hdfs_file)
a.saveAsTextFile(/some_other_hdfs_file, codec) fails with following
trace in Spark 1.3, works in Spark 1.2 in same env
15/04/14 18:06:15 INFO
Hi,
I am unable to access the metrics servlet on spark 1.2. I tried to access
it from the app master UI on port 4040 but i dont see any metrics there. Is
it a known issue with spark 1.2 or am I doing something wrong?
Also how do I publish my own metrics and view them on this servlet?
Thanks
Env - Spark 1.3 Hadoop 2.3, Kerbeos
xx.saveAsTextFile(path, codec) gives following trace. Same works with
Spark 1.2 in same environment
val codec = classOf[some codec class]
val a = sc.textFile(/some_hdfs_file)
a.saveAsTextFile(/some_other_hdfs_file, codec) fails with following trace
in Spark
Here is related problem:
http://apache-spark-user-list.1001560.n3.nabble.com/Launching-history-server-problem-td12574.html
but no answer.
What I'm trying to do: wrap spark-history with /etc/init.d script
Problems I have: can't make it read spark-defaults.conf
I've put this file here:
:0.7.0:compile
[INFO] | | \- javax.servlet:servlet-api:jar:2.5:compile
FYI
On Thu, Mar 26, 2015 at 3:36 PM, Pala M Muthaia
mchett...@rocketfuelinc.com wrote:
Hi,
We are trying to build spark 1.2 from source (tip of the branch-1.2 at
the moment). I tried to build spark using
-servlet*/artifactId
/dependency
Pretty much all of the missing class definition errors came up while
building HttpServer.scala, and went away after the above dependencies were
included.
My guess is official build for spark 1.2 is working already. My question
is what is wrong with my
Hi,
We are trying to build spark 1.2 from source (tip of the branch-1.2 at the
moment). I tried to build spark using the following command:
mvn -U -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive
-Phive-thriftserver -DskipTests clean package
I encountered various missing class definition
...@rocketfuelinc.com
wrote:
Hi,
We are trying to build spark 1.2 from source (tip of the branch-1.2 at the
moment). I tried to build spark using the following command:
mvn -U -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive
-Phive-thriftserver -DskipTests clean package
I encountered various
In 1.2 it's a member of SchemaRDD and it becomes available on RDD (through
the type class mechanism) when you add a SQLContext, like so.
val sqlContext = new SQLContext(sc)import sqlContext._
In 1.3, the method has moved to the new DataFrame type.
Dean Wampler, Ph.D.
Author: Programming Scala,
Hi,
I am running spark when I use sc.version I get 1.2 but when I call
registerTempTable(MyTable) I get error saying registedTempTable is not a
member of RDD
Why?
--
Eran | CTO
Hi,
I have received three replies to my question on my personal e-mail, why
don't they also show up on the mailing list? I would like to reply to the 3
users through a thread.
Thanks,
Maria
--
View this message in context:
Thanks.
I am new to the environment and running cloudera CDH5.3 with spark in it.
apparently when running in spark-shell this command val sqlContext = new
SQLContext(sc)
I am failing with the not found type SQLContext
Any idea why?
On Mon, Mar 23, 2015 at 3:05 PM, Dean Wampler
Have you tried adding the following ?
import org.apache.spark.sql.SQLContext
Cheers
On Mon, Mar 23, 2015 at 6:45 AM, IT CTO goi@gmail.com wrote:
Thanks.
I am new to the environment and running cloudera CDH5.3 with spark in it.
apparently when running in spark-shell this command val
In this thread:
http://search-hadoop.com/m/JW1q5DM69G
I only saw two replies. Maybe some people forgot to use 'Reply to All' ?
Cheers
On Mon, Mar 23, 2015 at 8:19 AM, mrm ma...@skimlinks.com wrote:
Hi,
I have received three replies to my question on my personal e-mail, why
don't they also
Hi,
I recently changed from Spark 1.1. to Spark 1.2., and I noticed that it
loses all executors whenever I have any Python code bug (like looking up a
key in a dictionary that does not exist). In earlier versions, it would
raise an exception but it would not lose all executors.
Anybody
Isn't that a feature? Other than running a buggy pipeline, just kills all
executors? You can always handle exceptions with proper try catch in your
code though.
Thanks
Best Regards
On Fri, Mar 20, 2015 at 3:51 PM, mrm ma...@skimlinks.com wrote:
Hi,
I recently changed from Spark 1.1. to Spark
Maybe this is related to a bug in 1.2 [1], it's fixed in 1.2.2 (not
released), could checkout the 1.2 branch and verify that?
[1] https://issues.apache.org/jira/browse/SPARK-5788
On Fri, Mar 20, 2015 at 3:21 AM, mrm ma...@skimlinks.com wrote:
Hi,
I recently changed from Spark 1.1. to Spark
Hi SM,
Apologize for delayed response.
No, the issue is with Spark 1.2.0. There is a bug in Spark 1.2.0.
Recently Spark have latest 1.3.0 release so it might have fixed in it.
I am not planning to test it soon, may be after some time.
You can try for it.
Regards,
Shailesh
--
View this
I'm using a pre-built Spark; I'm not trying to compile Spark.
The compile error appears when I try to register HighlyCompressedMapStatus
in my program:
kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
If I don't register it, I get a runtime error saying that it needs
Giving a bit more detail on the error would make it a lot easier for others
to help you out. Eg., in this case, it would have helped if included your
actual compile error.
In any case, I'm assuming your issue is b/c that class if private to
spark. You can sneak around that by using
The error is in the original post.
Here's the recipe that worked for me:
kryo.register(Class.forName(org.roaringbitmap.RoaringArray$Element))
kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
kryo.register(classOf[Array[Short]])
I'm not sure what you mean. Are you asking how you can recompile all of
spark and deploy it, instead of using one of the pre-built versions?
https://spark.apache.org/docs/latest/building-spark.html
Or are you seeing compile problems specifically w/
HighlyCompressedMapStatus? The code
Does anyone know how to get the HighlyCompressedMapStatus to compile?
I will try turning off kryo in 1.2.0 and hope things don't break. I want
to benefit from the MapOutputTracker fix in 1.2.0.
On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid iras...@cloudera.com wrote:
the scala syntax for
the scala syntax for arrays is Array[T], not T[], so you want to use
something:
kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
kryo.register(classOf[Array[Short]])
nonetheless, the spark should take care of this itself. I'll look into it
later today.
On Mon, Mar 2, 2015
I think this is a Java vs scala syntax issue. Will check.
On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra arun.lut...@gmail.com wrote:
Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
I tried this as a workaround:
import org.apache.spark.scheduler._
import
Can someone confirm if they can run UDFs in group by in spark1.2?
I have two builds running -- one from a custom build from early December
(commit 4259ca8dd12) which works fine, and Spark1.2-RC2.
On the latter I get:
jdbc:hive2://XXX.208:10001 select
Seems you hit https://issues.apache.org/jira/browse/SPARK-4296. It has been
fixed in 1.2.1 and 1.3.
On Thu, Feb 26, 2015 at 1:22 PM, Yana Kadiyska yana.kadiy...@gmail.com
wrote:
Can someone confirm if they can run UDFs in group by in spark1.2?
I have two builds running -- one from a custom
Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
I tried this as a workaround:
import org.apache.spark.scheduler._
import org.roaringbitmap._
...
kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
kryo.register(classOf[org.roaringbitmap.RoaringArray])
Hi
We're using Spark in our app's unit tests. The tests start spark
context with local[*] and test time now is 178 seconds on spark 1.2
instead of 41 seconds on 1.0.
We are using spark version from cloudera CDH (1.2.0-cdh5.3.1).
Could you give some hints what could cause that? and where
I think Xuefeng Wu's suggestion is likely correct. This different is more
likely explained by the compression library changing versions than sort vs
hash shuffle (which should not affect output size significantly). Others
have reported that switching to lz4 fixed their issue.
We should document
:14 (GMT+09:00)
*Title* : Re: Shuffle write increases in spark 1.2
If you have a small reproduction for this issue, can you open a ticket at
https://issues.apache.org/jira/browse/SPARK ?
On December 29, 2014 at 7:10:02 PM, Kevin Jung (itsjb.j...@samsung.com)
wrote:
Hi all,
The size
I double check the 1.2 feature list and found out that the new sort-based
shuffle manager has nothing to do with HashPartitioner :- Sorry for the
misinformation.
In another hand. This may explain increase in shuffle spill as a side effect
of the new shuffle manager, let me revert
Same problem here, shuffle write increased from 10G to over 64G, since I'm
running on amazon EC2 this always cause temporary folder to consume all the
disk space. Still looking for a solution.
BTW, the 64G shuffle write is encountered on shuffling a pairRDD with
HashPartitioner, so its not
Hello,
as the original message never got accepted to the mailinglist, I quote it
here completely:
Kevin Jung wrote
Hi all,
The size of shuffle write showing in spark web UI is much different when I
execute same spark job on same input data(100GB) in both spark 1.1 and
spark 1.2
1.2.
At the same sortBy stage, the size of shuffle write is 39.7GB in spark 1.1
but 91.0GB in spark 1.2.
I set spark.shuffle.manager option to hash because it's default value is
changed but spark 1.2 writes larger file than spark 1.1.
Can anyone tell me why this happens?
Thanks
Kevin
I'm
Is this what you are looking for
1. Build Spark with the YARN profile
http://spark.apache.org/docs/1.2.0/building-spark.html. Skip this step
if you are using a pre-packaged distribution.
2. Locate the spark-version-yarn-shuffle.jar. This should be under
1.2 writing on parquet after a join never ends - GC problems
Hi all,
I’m experiencing a strange behaviour of spark 1.2.
I’ve a 3 node cluster + the master.
each node has:
1 HDD 7200 rpm 1 TB
16 GB RAM
8 core
I configured executors with 6 cores and 10 GB each (
spark.storage.memoryFraction = 0.6
To: Praveen Garg praveen.g...@guavus.commailto:praveen.g...@guavus.com
Cc: user@spark.apache.orgmailto:user@spark.apache.org
user@spark.apache.orgmailto:user@spark.apache.org
Subject: Re: Shuffle read/write issue in spark 1.2
Even I observed the same issue.
On Fri, Feb 6, 2015 at 12:19 AM, Praveen
...@guavus.com
Cc: user@spark.apache.org user@spark.apache.org
Subject: Re: Shuffle read/write issue in spark 1.2
Even I observed the same issue.
On Fri, Feb 6, 2015 at 12:19 AM, Praveen Garg praveen.g...@guavus.com
wrote:
Hi,
While moving from spark 1.1 to spark 1.2, we are facing an issue
Yes. It improved the performance but not only with spark 1.2 but spark 1.1
also. Precisely, job took more time to run in spark 1.2 with default options
but got completed in almost equal time when ran with “lz4” as of spark 1.1 with
“lz4”.
From: Aaron Davidson ilike...@gmail.commailto:ilike
Hi all,
I’m experiencing a strange behaviour of spark 1.2.
I’ve a 3 node cluster + the master.
each node has:
1 HDD 7200 rpm 1 TB
16 GB RAM
8 core
I configured executors with 6 cores and 10 GB each (
spark.storage.memoryFraction = 0.6 )
My job is pretty simple:
val file1 = sc.parquetFile
Even I observed the same issue.
On Fri, Feb 6, 2015 at 12:19 AM, Praveen Garg praveen.g...@guavus.com
wrote:
Hi,
While moving from spark 1.1 to spark 1.2, we are facing an issue where
Shuffle read/write has been increased significantly. We also tried running
the job by rolling back
/browse/SPARK-5081
--- *Original Message* ---
*Sender* : Josh Rosenrosenvi...@gmail.com
*Date* : 2015-01-05 06:14 (GMT+09:00)
*Title* : Re: Shuffle write increases in spark 1.2
If you have a small reproduction for this issue, can you open a ticket at
https://issues.apache.org/jira
We found the problem and already fixed it. Basically, spark-ec2 requires ec2
instances to have external ip addresses. You need to specify this in the ASW
console.
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
fixed it. Basically, spark-ec2
requires ec2 instances to have external ip addresses. You need to specify
this in the ASW console.
--
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
To: charles.fed
addresses. You need to specify
this in the ASW console.
--
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
To: charles.fed...@gmail.com; pzybr...@gmail.com; eyc...@hotmail.com
CC: user
...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
To: charles.fed...@gmail.com; pzybr...@gmail.com; eyc...@hotmail.com
CC: user@spark.apache.org
For those who found that absolute vs. relative path for the pem file
mattered, what OS and shell
instances to have external ip addresses. You need to specify
this in the ASW console.
--
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
To: charles.fed...@gmail.com; pzybr...@gmail.com; eyc
addresses. You need to
specify
this in the ASW console.
--
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
To: charles.fed...@gmail.com; pzybr...@gmail.com; eyc...@hotmail.com
CC: user
to specify
this in the ASW console.
--
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
To: charles.fed...@gmail.com; pzybr...@gmail.com; eyc...@hotmail.com
CC: user@spark.apache.org
For those who
an absolute path to the pem file
On Jan 26, 2015, at 8:57 PM, ey-chih chow eyc...@hotmail.com wrote:
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit
I've read that this is supposed to be a rather significant optimization to
the shuffle system in 1.1.0 but I'm not seeing much documentation on
enabling this in Yarn. I see github classes for it in 1.2.0 and a property
spark.shuffle.service.enabled in the spark-defaults.conf.
The code mentions
path to the pem file
On Jan 26, 2015, at 8:57 PM, ey-chih chow eyc...@hotmail.com wrote:
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit/5dd8458d2ab
where but I have a
memory of seeing somewhere that the AMI was only in us-east
cheers
On Mon, Jan 26, 2015 at 8:47 PM, Håkan Jonsson haj...@gmail.com wrote:
Thanks,
I also use Spark 1.2 with prebuilt for Hadoop 2.4. I launch both 1.1 and
1.2 with the same command:
./spark-ec2 -k foo -i
Try using an absolute path to the pem file
On Jan 26, 2015, at 8:57 PM, ey-chih chow eyc...@hotmail.com wrote:
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit/5dd8458d2ab9753aae939b3bb33be953e2c13a70
But the script was still hung at the following message:
Waiting for cluster to enter 'ssh-ready
:
Using Spark 1.2
Read a CSV file, apply schema to convert to SchemaRDD and then
schemaRdd.saveAsParquetFile
If the schema includes Timestamptype, it gives following trace when
doing the save
Exception in thread main java.lang.RuntimeException: Unsupported
datatype TimestampType
Thanks. But after setting spark.shuffle.blockTransferService to nio
application fails with Akka Client disassociation.
15/01/27 13:38:11 ERROR TaskSchedulerImpl: Lost executor 3 on
wynchcs218.wyn.cnw.co.nz: remote Akka client disassociated
15/01/27 13:38:11 INFO TaskSetManager: Re-queueing tasks
,
I also use Spark 1.2 with prebuilt for Hadoop 2.4. I launch both 1.1 and
1.2 with the same command:
./spark-ec2 -k foo -i bar.pem launch mycluster
By default this launches in us-east-1. I tried changing the the region
using:
-r us-west-1 but that had the same result:
Could not resolve AMI
Thanks,
I also use Spark 1.2 with prebuilt for Hadoop 2.4. I launch both 1.1 and
1.2 with the same command:
./spark-ec2 -k foo -i bar.pem launch mycluster
By default this launches in us-east-1. I tried changing the the region
using:
-r us-west-1 but that had the same result:
Could not resolve
I definitely have Spark 1.2 running within EC2 using the spark-ec2 scripts.
I downloaded Spark 1.2 with prebuilt for Hadoop 2.4 and later.
What parameters are you using when you execute spark-ec2?
I am launching in the us-west-1 region (ami-7a320f3f) which may explain
things.
On Mon Jan 26 2015
/LogicalTypes.md),
any reason why Date / Timestamp are not supported right now ?
Thanks,
Manoj
On Fri, Jan 23, 2015 at 11:40 AM, Manoj Samel manojsamelt...@gmail.com
wrote:
Using Spark 1.2
Read a CSV file, apply schema to convert to SchemaRDD and then
schemaRdd.saveAsParquetFile
On Fri, Jan 23, 2015 at 11:40 AM, Manoj Samel manojsamelt...@gmail.com
wrote:
Using Spark 1.2
Read a CSV file, apply schema to convert to SchemaRDD and then
schemaRdd.saveAsParquetFile
If the schema includes Timestamptype, it gives following trace when
doing the save
Exception in thread main
Hi,
When I try to launch a standalone cluster on EC2 using the scripts in the
ec2 directory for Spark 1.2, I get the following error:
Could not resolve AMI at:
https://raw.github.com/mesos/spark-ec2/v4/ami-list/us-east-1/pvm
It seems there is not yet any AMI available on EC2. Any ideas when
This was a regression caused by Netty Block Transfer Service. The fix for
this just barely missed the 1.2 release, and you can see the associated
JIRA here: https://issues.apache.org/jira/browse/SPARK-4837
Current master has the fix, and the Spark 1.2.1 release will have it
included. If you don't
Can anyone please let me know ?
I don't want to open all ports on n/w. So, am interested in the property by
which this new port I can configure.
Shailesh
--
View this message in context:
/LogicalTypes.md),
any reason why Date / Timestamp are not supported right now ?
Thanks,
Manoj
On Fri, Jan 23, 2015 at 11:40 AM, Manoj Samel manojsamelt...@gmail.com
wrote:
Using Spark 1.2
Read a CSV file, apply schema to convert to SchemaRDD and then
schemaRdd.saveAsParquetFile
If the schema
change related to serialize the closure cause LogParser is
not a singleton any more, then it is initialized for every task.
Could you change it to a Broadcast?
On Tue, Jan 20, 2015 at 10:39 PM, Fengyun RAO raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2
Using Spark 1.2
Read a CSV file, apply schema to convert to SchemaRDD and then
schemaRdd.saveAsParquetFile
If the schema includes Timestamptype, it gives following trace when doing
the save
Exception in thread main java.lang.RuntimeException: Unsupported datatype
TimestampType
Seems like it is a bug rather than a feature.
I filed a bug report: https://issues.apache.org/jira/browse/SPARK-5363
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-1-slow-working-Spark-1-2-fast-freezing-tp21278p21317.html
Sent from the Apache Spark
We have not meet this issue, so not sure there are bugs related to
reused worker or not.
Could provide more details about it?
On Wed, Jan 21, 2015 at 2:27 AM, critikaled isasmani@gmail.com wrote:
I'm also facing the same issue.
is this a bug?
--
View this message in context:
it to a Broadcast?
On Tue, Jan 20, 2015 at 10:39 PM, Fengyun RAO raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2, but found
the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole
year data,
quite stable
, 2015 at 10:39 PM, Fengyun RAO raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole year
data,
quite stable.
the main script
recently tried to migrate from Spark 1.1 to Spark 1.2 - using
PySpark. Initially, I was super glad, noticing that Spark 1.2 is way
faster
than Spark 1.1. However, the initial joy faded quickly when I noticed
that
all my stuff didn't successfully terminate operations anymore. Using
, Jan 20, 2015 at 9:00 PM, TJ Klein tjkl...@gmail.com wrote:
Hi,
I just recently tried to migrate from Spark 1.1 to Spark 1.2 - using
PySpark. Initially, I was super glad, noticing that Spark 1.2 is way
faster
than Spark 1.1. However, the initial joy faded quickly when I noticed
tjkl...@gmail.com wrote:
Hi,
I just recently tried to migrate from Spark 1.1 to Spark 1.2 -
using
PySpark. Initially, I was super glad, noticing that Spark 1.2 is
way
faster
than Spark 1.1. However, the initial joy faded quickly when I
noticed
that
all my stuff
Hello,
Recently, I have upgraded my setup to Spark 1.2 from Spark 1.1.
I have 4 node Ubuntu Spark Cluster.
With Spark 1.1, I used to write Spark Scala program in Eclipse on my Windows
development host and submit the job on Ubuntu Cluster, from Eclipse (Windows
machine).
As on my network not all
I'm also facing the same issue.
is this a bug?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-1-slow-working-Spark-1-2-fast-freezing-tp21278p21283.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole year
data,
quite stable.
the main script is as below
sc.textFile
:
Currently we are migrating from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole year
data,
quite stable.
the main script is as below
sc.textFile(inputPath)
.flatMap(line
web UI because itcan bepossibly related to my case.
Thanks
Kevin
--- Original Message ---
Sender : Fengyun RAOraofeng...@gmail.com
Date : 2015-01-21 17:41 (GMT+09:00)
Title : Re: spark 1.2 three times slower than spark 1.1, why?
maybe you mean different spark-submit script
from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole
year data,
quite stable.
the main script is as below
sc.textFile(inputPath)
.flatMap(line = LogParser.parseLine(line
* : Re: spark 1.2 three times slower than spark 1.1, why?
maybe you mean different spark-submit script?
we also use the same spark-submit script, thus the same memory, cores,
etc configuration.
2015-01-21 15:45 GMT+08:00 Sean Owen so...@cloudera.com:
I don't know of any reason to think
task.
Could you change it to a Broadcast?
On Tue, Jan 20, 2015 at 10:39 PM, Fengyun RAO raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2, but found
the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed
it to a Broadcast?
On Tue, Jan 20, 2015 at 10:39 PM, Fengyun RAO raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole year
data,
quite
at 10:39 PM, Fengyun RAO raofeng...@gmail.com
wrote:
Currently we are migrating from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole year
data,
quite stable.
the main script is as below
Please also see this thread:
http://search-hadoop.com/m/JW1q5De7pU1
On Tue, Jan 20, 2015 at 3:58 PM, Sean Owen so...@cloudera.com wrote:
Guava is shaded in Spark 1.2+. It looks like you are mixing versions
of Spark then, with some that still refer to unshaded Guava. Make sure
you
Guava is shaded in Spark 1.2+. It looks like you are mixing versions
of Spark then, with some that still refer to unshaded Guava. Make sure
you are not packaging Spark with your app and that you don't have
other versions lying around.
On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari sbirar
Hello,
I recently upgraded my setup from Spark 1.1 to Spark 1.2.
My existing applications are working fine on ubuntu cluster.
But, when I try to execute Spark MLlib application from Eclipse (Windows
node) it gives java.lang.NoClassDefFoundError:
com/google/common/base/Preconditions exception
double checked the libraries. I am linking only with Spark 1.2.
Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked
and nothing else.
Thanks,
Shailesh
On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen so...@cloudera.com wrote:
Guava is shaded in Spark 1.2+. It looks like you
...@berkeley.edu
fnoth...@eecs.berkeley.edu
202-340-0466
On Jan 20, 2015, at 5:13 PM, Shailesh Birari sbirar...@gmail.com
wrote:
Hello,
I double checked the libraries. I am linking only with Spark 1.2.
Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked
and nothing else
...@gmail.com wrote:
Hello,
I double checked the libraries. I am linking only with Spark 1.2.
Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked and
nothing else.
Thanks,
Shailesh
On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen so...@cloudera.com wrote:
Guava is shaded in Spark
Hi,
I just recently tried to migrate from Spark 1.1 to Spark 1.2 - using
PySpark. Initially, I was super glad, noticing that Spark 1.2 is way faster
than Spark 1.1. However, the initial joy faded quickly when I noticed that
all my stuff didn't successfully terminate operations anymore. Using
Hello,
I double checked the libraries. I am linking only with Spark 1.2.
Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked and
nothing else.
Thanks,
Shailesh
On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen so...@cloudera.com wrote:
Guava is shaded in Spark 1.2+. It looks
1 - 100 of 188 matches
Mail list logo
