I would recommend to use the JedisPool with autocloseable pattern:
private JedisPool pool = new JedisPool(host, port);
try (Jedis jedis = pool.getResource())
{
/*do magic to jedis*/
}
pool.destroy();
We use this contruct successfully in a foreachPartition action.
Am 24.03.2016 um 15:20 schrieb Michel Hubert:
>
> No.
>
>
>
> But I may be on to something.
>
> I use Jedis to send data to Redis.
>
>
>
> I used a ThreadLocal construct:
>
>
>
> private static final ThreadLocal<Jedis> /jedis /= new
> ThreadLocal<Jedis>(){
> @Override
> protected Jedis initialValue()
> {
> return new Jedis("10.101.41.19",6379);
> }
> };
>
>
>
> and then
>
>
>
> .foreachRDD(new VoidFunction<JavaRDD<TopData>>() { public void
> call(JavaRDD<TopData> rdd) throws Exception { for (TopData t:
> rdd.take(top)) { jedis … }
>
>
>
> May this resulted in a memory leak?
>
>
>
> *Van:*Ted Yu [mailto:[email protected]]
> *Verzonden:* donderdag 24 maart 2016 15:15
> *Aan:* Michel Hubert <[email protected]>
> *CC:* [email protected]
> *Onderwerp:* Re: apache spark errors
>
>
>
> Do you have history server enabled ?
>
>
>
> Posting your code snippet would help us understand your use case (and
> reproduce the leak).
>
>
>
> Thanks
>
>
>
> On Thu, Mar 24, 2016 at 6:40 AM, Michel Hubert <[email protected]
> <mailto:[email protected]>> wrote:
>
> <dependencies>
> <dependency> <!-- Spark dependency -->
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.10</artifactId>
> <version>1.6.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming_2.10</artifactId>
> <version>1.6.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming-kafka_2.10</artifactId>
> <version>1.6.1</version>
> </dependency>
>
> <dependency>
> <groupId>org.elasticsearch</groupId>
> <artifactId>elasticsearch</artifactId>
> <version>2.2.0</version>
> </dependency>
>
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.10</artifactId>
> <version>0.8.2.2</version>
> </dependency>
>
>
> <dependency>
> <groupId>org.elasticsearch</groupId>
> <artifactId>elasticsearch-spark_2.10</artifactId>
> <version>2.2.0</version>
> </dependency>
> <dependency>
> <groupId>redis.clients</groupId>
> <artifactId>jedis</artifactId>
> <version>2.8.0</version>
> <type>jar</type>
> <scope>compile</scope>
> </dependency>
> </dependencies>
>
>
>
>
>
> How can I look at those tasks?
>
>
>
> *Van:*Ted Yu [mailto:[email protected]
> <mailto:[email protected]>]
> *Verzonden:* donderdag 24 maart 2016 14:33
> *Aan:* Michel Hubert <[email protected] <mailto:[email protected]>>
> *CC:* [email protected] <mailto:[email protected]>
> *Onderwerp:* Re: apache spark errors
>
>
>
> Which release of Spark are you using ?
>
>
>
> Have you looked the tasks whose Ids were printed to see if there
> was more clue ?
>
>
>
> Thanks
>
>
>
> On Thu, Mar 24, 2016 at 6:12 AM, Michel Hubert <[email protected]
> <mailto:[email protected]>> wrote:
>
> HI,
>
>
>
> I constantly get these errors:
>
>
>
> 0 [Executor task launch worker-15] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 6564500 bytes, TID = 38969
>
> 310002 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5523550 bytes, TID = 43270
>
> 318445 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 6879566 bytes, TID = 43408
>
> 388893 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5572546 bytes, TID = 44382
>
> 418186 [Executor task launch worker-13] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5289222 bytes, TID = 44795
>
> 488421 [Executor task launch worker-4] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 8738142 bytes, TID = 45769
>
> 619276 [Executor task launch worker-4] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5759312 bytes, TID = 47571
>
> 632275 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5602240 bytes, TID = 47709
>
> 644989 [Executor task launch worker-13] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5326260 bytes, TID = 47863
>
> 720701 [Executor task launch worker-12] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5399578 bytes, TID = 48959
>
> 1147961 [Executor task launch worker-16] ERROR
> org.apache.spark.executor.Executor - Managed memory leak
> detected; size = 5251872 bytes, TID = 54922
>
>
>
>
>
> How can I fix this?
>
>
>
> With kind regard,
>
>
>
> Michel
>
>
>
>
>
--
*Max Schmidt, Senior Java Developer* | [email protected]
<mailto:[email protected]> | LinkedIn
<https://www.linkedin.com/in/maximilian-schmidt-9893b7bb/>
Datapath.io
Decreasing AWS latency.
Your traffic optimized.
Datapath.io GmbH
Mainz | HRB Nr. 46222
Sebastian Spies, CEO
