Alec, For writing to a database in Trident, I would suggest implementing a Trident State. That will allow you to process batches of tuples in bulk. Trident functions operate on individual tuples, so writing to database in a function will result in one insert for every tuple (inefficient).
You can find more information on Trident State here: https://storm.apache.org/documentation/Trident-state -Taylor On Dec 5, 2014, at 5:51 PM, Sa Li <[email protected]> wrote: > Hi, Dmytro > > Thanks for the reply, I am using trident topology, here is my static class, > but I think this is incorrect since trident process data in batch, but > following code process single row. > > public static class WriteDB extends BaseFunction { > private Connection conn = null ; > PreparedStatement ps = null; > > @Override > public final void execute(final TridentTuple tuple, final > TridentCollector collector) { > int user = tuple.getInteger(0); > String value = tuple.getString(1); > final StringBuilder queryBuilder = new StringBuilder() > .append("INSERT INTO test.state(userid, event) > VALUES(") > .append(user) > .append(", '") > .append(value) > .append("')"); > System.out.println(queryBuilder.toString()); > try { > ps = > conn.prepareStatement(queryBuilder.toString()) ; > ps.execute(); > collector.emit(new > Values(tuple.getStringByField("event"))); > } > catch (SQLException ex) { > System.err.println("Caught IOException: " + > ex.getMessage()); > } finally { > if (ps != null) { > try { > > ps.close(); > } catch > (SQLException ex) { > } > } > } > } > } > > > Any idea? > > thanks > > On Fri, Dec 5, 2014 at 2:35 PM, Dima Dragan <[email protected]> wrote: > Hi, > > I think better approach is creating static class with thread-safe singleton > initialization of connection pool (for example, I use HikariCP for SQL db and > JedisPool for Redis) and provide public access for getting connection from it > to bolt. Init pool in prepare method, take connection,execute smth, give it > back in execute method. > > So every worker will get it's own pool which is shared between bolts. > > Best regards, > Dmytro Dragan > > On Dec 5, 2014 11:57 PM, "Sa Li" <[email protected]> wrote: > Hi, all > > Right now I am able to write tuples into a file, and use copy command to load > into database, but this is obviously not a perfect solution, increase > complexity and overheads. Any idea to make it simpler? > > > thanks > > > Alec >
signature.asc
Description: Message signed with OpenPGP using GPGMail
