Shivpriya, I have thought about solving that same problem, and there is no easy answer in the current code that I could see. You would have to modify hdfs-bolt to track a group of file handles into HDFS and define a mechanism to map a tuple to the corresponding file handle, create it if it doesn't already exist, etc. And you have to do it in a way that works for all of the existing flavors of hdfs-bolt (text files, sequence files, avro files). Now hive-bolt does something conceptually similar in that in manages a connection pool, limits maximum connections, closes old connections, etc. so the pattern does exist but implementing it would be tantamount to a rewrite of hdfs-bolt.
For the problem I was trying to solve, this approach would have provided only a very small benefit, so I didn't pursue it beyond some high level design work. You could open a JIRA for this feature request. Thanks, Aaron From: Shivpriya Tamboskar <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, January 4, 2016 at 3:40 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Storm-HDFS bolt file paths according to input Tuples Hello, I am trying to write data into HDFS using the storm-hdfs bolt http://storm.apache.org/documentation/storm-hdfs.html The problem I am trying to solve is that I want the file paths to be set according to the input tuples. FileNameFormat fileNameFormat = new DefaultFileNameFormat() .withPath("/foo/"); I am getting inputs from various users and what want to do is set the file paths according to the username e.g. "/user1/" , "/user2/" As far as I know once the path is set for the bolt it can't be changed unless you have file rotation policy set which is not what I want to do. Has anyone else tried to solve this before ? Any suggestions would be appreciated. Thanks Shivpriya
