Can you instantiate Jedis class in a standalone program and see its memory
footprint ?
Just to get some idea how much its cost is.
Also, can you look for other approach which doesn't require using
ThreadLocal ?
On Thu, Mar 24, 2016 at 7:20 AM, Michel Hubert <mich...@phact.nl> wrote:
> No.
>
>
>
> But I may be on to something.
>
> I use Jedis to send data to Redis.
>
>
>
> I used a ThreadLocal construct:
>
>
>
> private static final ThreadLocal<Jedis> *jedis *= new
> ThreadLocal<Jedis>(){
> @Override
> protected Jedis initialValue()
> {
> return new Jedis("10.101.41.19",6379);
> }
> };
>
>
>
> and then
>
>
>
> .foreachRDD(new VoidFunction<JavaRDD<TopData>>() {
> public void call(JavaRDD<TopData> rdd) throws Exception {
>
> for (TopData t: rdd.take(top)) {
> jedis …
> }
>
>
>
> May this resulted in a memory leak?
>
>
>
> *Van:* Ted Yu [mailto:yuzhih...@gmail.com]
> *Verzonden:* donderdag 24 maart 2016 15:15
>
> *Aan:* Michel Hubert <mich...@phact.nl>
> *CC:* user@spark.apache.org
> *Onderwerp:* Re: apache spark errors
>
>
>
> Do you have history server enabled ?
>
>
>
> Posting your code snippet would help us understand your use case (and
> reproduce the leak).
>
>
>
> Thanks
>
>
>
> On Thu, Mar 24, 2016 at 6:40 AM, Michel Hubert <mich...@phact.nl> wrote:
>
> <dependencies>
> <dependency> <!-- Spark dependency -->
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.10</artifactId>
> <version>1.6.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming_2.10</artifactId>
> <version>1.6.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming-kafka_2.10</artifactId>
> <version>1.6.1</version>
> </dependency>
>
> <dependency>
> <groupId>org.elasticsearch</groupId>
> <artifactId>elasticsearch</artifactId>
> <version>2.2.0</version>
> </dependency>
>
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.10</artifactId>
> <version>0.8.2.2</version>
> </dependency>
>
>
> <dependency>
> <groupId>org.elasticsearch</groupId>
> <artifactId>elasticsearch-spark_2.10</artifactId>
> <version>2.2.0</version>
> </dependency>
> <dependency>
> <groupId>redis.clients</groupId>
> <artifactId>jedis</artifactId>
> <version>2.8.0</version>
> <type>jar</type>
> <scope>compile</scope>
> </dependency>
> </dependencies>
>
>
>
>
>
> How can I look at those tasks?
>
>
>
> *Van:* Ted Yu [mailto:yuzhih...@gmail.com]
> *Verzonden:* donderdag 24 maart 2016 14:33
> *Aan:* Michel Hubert <mich...@phact.nl>
> *CC:* user@spark.apache.org
> *Onderwerp:* Re: apache spark errors
>
>
>
> Which release of Spark are you using ?
>
>
>
> Have you looked the tasks whose Ids were printed to see if there was more
> clue ?
>
>
>
> Thanks
>
>
>
> On Thu, Mar 24, 2016 at 6:12 AM, Michel Hubert <mich...@phact.nl> wrote:
>
> HI,
>
>
>
> I constantly get these errors:
>
>
>
> 0 [Executor task launch worker-15] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 6564500 bytes, TID = 38969
>
> 310002 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5523550 bytes, TID = 43270
>
> 318445 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 6879566 bytes, TID = 43408
>
> 388893 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5572546 bytes, TID = 44382
>
> 418186 [Executor task launch worker-13] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5289222 bytes, TID = 44795
>
> 488421 [Executor task launch worker-4] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 8738142 bytes, TID = 45769
>
> 619276 [Executor task launch worker-4] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5759312 bytes, TID = 47571
>
> 632275 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5602240 bytes, TID = 47709
>
> 644989 [Executor task launch worker-13] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5326260 bytes, TID = 47863
>
> 720701 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5399578 bytes, TID = 48959
>
> 1147961 [Executor task launch worker-16] ERROR
> org.apache.spark.executor.Executor - Managed memory leak detected; size =
> 5251872 bytes, TID = 54922
>
>
>
>
>
> How can I fix this?
>
>
>
> With kind regard,
>
>
>
> Michel
>
>
>
>
>
