Unsubscribe

2016-06-25 Thread Shivpriya Tamboskar 
Unsubscribe

Sent from my iPhone

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

java.nio.channels.ClosedChannelException

2016-03-04 Thread Shivpriya Tamboskar 
Hello community,
 I am running a simple consumer for kafka such as this

   int timeout = 8;
   int bufferSize = 64*1024;
   consumer = new SimpleConsumer(host, port,timeout, bufferSize, clientName);

This runs fine for a couple for hours but I get an exception later on 
kafka.consumer.SimpleConsumer: Reconnect due to socket error: 
java.nio.channels.ClosedChannelException

and consumer stops ... has anyone faced this problem before ?

Thanks
Shivpriya

Re: Storm-HDFS bolt file paths according to input Tuples

2016-01-04 Thread Shivpriya Tamboskar 
Thank you Aaron,
 I too did try to check the code to see if there is any way to modify the code 
and get this feature.
But it seemed too intricate. I was trying to get an idea if there are any 
possible solutions  the other users came up with in this situation.

-Shivpriya

From: "Aaron.Dossett" 
mailto:aaron.doss...@target.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
mailto:user@storm.apache.org>>
Date: Monday, January 4, 2016 at 1:49 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
mailto:user@storm.apache.org>>
Subject: Re: Storm-HDFS bolt file paths according to input Tuples

Shivpriya,

I have thought about solving that same problem, and there is no easy answer in 
the current code that I could see.  You would have to modify hdfs-bolt to track 
a group of file handles into HDFS and define a mechanism to map a tuple to the 
corresponding file handle, create it if it doesn’t already exist, etc.  And you 
have to do it in a way that works for all of the existing flavors of hdfs-bolt 
(text files, sequence files, avro files).  Now hive-bolt does something 
conceptually similar in that in manages a connection pool, limits maximum 
connections, closes old connections, etc. so the pattern does exist but 
implementing it would be tantamount to a rewrite of hdfs-bolt.

For the problem I was trying to solve, this approach would have provided only a 
very small benefit, so I didn’t pursue it beyond some high level design work.  
You could open a JIRA for this feature request.

Thanks, Aaron

From: Shivpriya Tamboskar 
mailto:shivpriya_tambos...@symantec.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
mailto:user@storm.apache.org>>
Date: Monday, January 4, 2016 at 3:40 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
mailto:user@storm.apache.org>>
Subject: Storm-HDFS bolt file paths according to input Tuples

Hello,
  I am trying to write data into HDFS using the storm-hdfs bolt 
http://storm.apache.org/documentation/storm-hdfs.html

The problem I am trying to solve is that I want the file paths to be set 
according to the input tuples.


FileNameFormat fileNameFormat = new DefaultFileNameFormat()

.withPath("/foo/“);


I am getting inputs from various users and what want to do is set the file 
paths according to the username e.g.  "/user1/“ , “/user2/“


As far as I know once the path is set for the bolt it can’t be changed unless 
you have file rotation policy set which is not what I want to do.


Has anyone else tried to solve this before ? Any suggestions would be 
appreciated.


Thanks

Shivpriya

Storm-HDFS bolt file paths according to input Tuples

2016-01-04 Thread Shivpriya Tamboskar 
Hello,
  I am trying to write data into HDFS using the storm-hdfs bolt 
http://storm.apache.org/documentation/storm-hdfs.html

The problem I am trying to solve is that I want the file paths to be set 
according to the input tuples.


FileNameFormat fileNameFormat = new DefaultFileNameFormat()

.withPath("/foo/“);


I am getting inputs from various users and what want to do is set the file 
paths according to the username e.g.  "/user1/“ , “/user2/“


As far as I know once the path is set for the bolt it can’t be changed unless 
you have file rotation policy set which is not what I want to do.


Has anyone else tried to solve this before ? Any suggestions would be 
appreciated.


Thanks

Shivpriya