What is the size of your data, size of the cluster, are you using
spark-submit or an IDE, what spark version are you using?
Try spark-submit and increase the memory of the driver or the executors.
a.
On 22/3/19 17:19, KhajaAsmath Mohammed wrote:
Hi,
I am getting the below exception when
te all the
> productionName and product catagory together, without even consider removing
> duplication.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap
the stuff without String operations?
>
> On 24 September 2015 at 10:11, java8964 <java8...@hotmail.com
> <mailto:java8...@hotmail.com>> wrote:
> Try to increase partitions count, that will make each partition has less data.
>
> Yong
>
> Subject: Re: Java Heap Space
rtitioner of "userid".
Can you show us the query after you add "regex" and "concatenation"?
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 15:34:48 +0300
CC: user@spark.apache.org
To: jingyu.zh...@news.com.au; java8...@hotma
ns simply depends on the Hash partitioner of "userid".
>
> Can you show us the query after you add "regex" and "concatenation"?
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 15:34:48 +0300
&
ion.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 18:56:51 +0300
> CC: jingyu.
.
But both query still should push similar records count per partition, but with
much of different volume size of data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 18:56:51 +0300
CC: jingyu.zh...@news.com.au; user@spark.apache.org
To: java8...@hotmail.com
Yes
Yes, it’s possible. I use S3 as data source. My external tables has
partitioned. Belowed task is 193/200. Job has 2 stages and its 193. task of 200
in 2.stage because of sql.shuffle.partitions.
How can i avoid this situation, this is my query:
select userid,concat_ws('
Try to increase partitions count, that will make each partition has less data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 00:32:47 +0300
CC: user@spark.apache.org
To: java8...@hotmail.com
Yes, it’s possible. I use S3 as data source. My external
ta.
>
> Yong
>
> ----------
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 00:32:47 +0300
> CC: user@spark.apache.org
> To: java8...@hotmail.com
>
>
> Yes, it’s possible. I use S3 as data source. My extern
Any help?please.
Help me do a right configure.
李铖 lidali...@gmail.com于2015年4月7日星期二写道：
In my dev-test env .I have 3 virtual machines ,every machine have 12G
memory,8 cpu core.
Here is spark-defaults.conf,and spark-env.sh.Maybe some config is not
right.
I run this command :*spark-submit
-Dspark.deploy.defaultCores=. When you have not
enough memory, reduce the concurrency of your executor, it will lower the
memory requirement, with running in a slower speed.
Yong
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error occured when query 4G data file from HDFS
From: lidali
in a slower speed.
Yong
--
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error occured when query 4G data file from
HDFS
From: lidali...@gmail.com
To: user@spark.apache.org
Any help?please.
Help me do a right configure.
李铖 lidali
13 matches
Mail list logo
