NoSuchElementException

My code throws an exception when I am trying to create new DataSet from within SteamWriter sink Simplified version of the code val df = sparkSession.readStream .format("json") .option("nullValue", " ") .option("headerFlag", "true") .option("spark.sql.shuffle.partitions", 1)

Re: Possible memory leak after closing spark context in v2.0.1

52 PM, Sean Owen wrote: > Did you unpersist the broadcast objects? > > On Mon, Oct 17, 2016 at 10:02 AM lev wrote: > >> Hello, >> >> I'm in the process of migrating my application to spark 2.0.1, >> And I think there is some memory leaks related to Broadc

Possible memory leak after closing spark context in v2.0.1

Hello, I'm in the process of migrating my application to spark 2.0.1, And I think there is some memory leaks related to Broadcast joins. the application has many unit tests, and each individual test suite passes, but when running all together, it fails on OOM errors. In the begging of each suite

spark 2.0.0 - how to build an uber-jar?

hi, in spark 1.5, to build an uber-jar, I would just compile the code with: mvn ... package and that will create one big jar with all the dependencies. when trying to do the same with spark 2.0, I'm getting a tar.gz file instead. this is the full command I'm using: mvn -Pyarn -Phive -Phadoop-2.6

log4j custom appender ClassNotFoundException with spark 1.5.2

Hi, I'm using spark 1.5.2 and running on a yarn cluster and trying to use a custom log4j appender in my setup there are 3 jars: the uber jar: spark.yarn.jar=uber-jar.jar the jar that contains the main class: main.jar additional jar with dependencies: dep.jar (passed with the --jars flag to spark

Prevent spark from serializing some objects

nt to use my own serializer on production code Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Prevent-spark-from-serializing-some-objects-tp24700.html Sent from the Apache Spark User List mail

Using native blas with mllib

jar tf assembly\target\scala-2.10\spark-assembly-1.1.0-cdh5.2.0-hadoop2.5.0-cdh5.2.0.jar | grep "netlib-native_system-linux-x86_64.so" netlib-native_system-linux-x86_64.so netlib-native_system-linux-x86_64.so.asc Anything e

Re: Consistent hashing of RDD row

After checking the spark code, I now realize that an rdd that was cached to disk can't be evicted, so I will just persist the rdd to disk after the random numbers are created. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Consistent-hashing-of-RDD-row-tp20

Consistent hashing of RDD row

f the query get evicted and recalculated, will the index stay the same? I'd love to hear your thoughts on the matter. Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Consistent-hashing-of-RDD-row-tp20820.html Sent from the Apache Spark Us

repartition combined with zipWithIndex get stuck

938941 14/11/15 03:18:55 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 1096.0 B, free 132.5 MB) Am I doing something wrong here or is it a bug? Is there some work around? Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.na

Re: org/apache/commons/math3/random/RandomGenerator issue

appear in the compiled jar, but the error is still there. it's wired that the class is not found even when it's in the jar. - adding the math3 jar to a dir that in oozie.libpath. I'm running the spark jar with oozie, but that also didn't solve it. Thanks, Lev. -- View

Re: org/apache/commons/math3/random/RandomGenerator issue

"hdfs://" ) and it didn't work.. Thanks, Lev. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/org-apache-commons-math3-random-RandomGenerator-issue-tp15748p18453.html Sent from the Apache Spark User List mailing list archiv

Re: org/apache/commons/math3/random/RandomGenerator issue

Hi, I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having the same error. I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and it didn't help. Any ideas what might be the problem? Thanks, Lev. anny9699 wrote &