Parquet reads in Spark need lots of tempory heap memory due to
ColumnVectors and write block size. See a similar issue:
https://jira.snappydata.io/browse/SNAP-3111
In addition writes too consume significant amount of heap due to
parquet.block.size. One solution is to reduce the spark.executor.core
thods
using REST API from your application providing it the required
parameters like a remote procedure call.
Or you can try SnappyData that provides both of these (and much
more) out of the box.
Regards,
Sumedh Wale
SnappyData (http://www.snappyda
, Jul 22, 2017 at 6:19 AM, Sumedh
Wale <sw...@snappydata.io>
wrote:
On Saturday 22 July 2017 01:31 PM, kant kodali
wrote:
Is there a way to run Spark SQL throug
fuzzy joins). Its possible to do that using internal Spark
APIs but its not easy and you have to implement an efficient join
structure first. Or perhaps some existing libraries out there could work
for you (like https://github.com/soundcloud/cosine-lsh-join-spark?).
--
Sumedh Wale
SnappyData
On Saturday 22 July 2017 01:31 PM, kant kodali wrote:
Is there a way to run Spark SQL through REST?
There is spark-jobserver
(https://github.com/spark-jobserver/spark-jobserver). It does more than
just REST API (like long running SparkContext).
regards
--
Sumedh Wale
SnappyData (http
?
Thanks.
regards
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Thanks and Regards,
Weiping
regards
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
e.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-SSLEncryption
Regards,
Sourav
thanks
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.or
ing coalesce it will likely not use the full
processing power.
thanks
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional comma
.
Regards,
Gourav Sengupta
thanks
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
On Thu, Mar 3, 2016 at 12:33 PM, Sumedh
Wale <sw...@snappydata.io>
wrote:
aking so long time. So can
you suggest me the any optimized way, so i can reduce the
execution time.
My cluster has 3 slaves and 1 master.
Thanks.
sshfs) and mount the remote directory
locally. Since this uses FUSE, so works fine with normal user privileges.
Thanks,
Ben
Thanks
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
-
To unsubscribe, e-mail: use
So you can set SPARK_DIST_CLASSPATH in conf/spark-env.sh to add the
required jars (separated by platform's File.pathSeparator).
thanks
--
Sumedh Wale
SnappyData (http://www.snappydata.io)
Am 2. März 2016 um 17:05 schrieb Igor
B
13 matches
Mail list logo