Team, Is there any exactly once Hbase put operator ??
Also can we define two tables using the set table property for Hbase store ?? Thanks!! On Thursday, October 20, 2016, Jaspal Singh <[email protected]> wrote: > Thomas, > > When we try excluding the transitive dependencies using 'exclusion *', > getting the below error while running maven build: > > [ERROR] COMPILATION ERROR : > [INFO] ------------------------------------------------------------- > [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-mapr-streams/ > src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[51,18] > cannot access org.apache.hadoop.hbase.CellScannable > class file for org.apache.hadoop.hbase.CellScannable not found > [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-mapr-streams/ > src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[105,10] > cannot access org.apache.hadoop.hbase.io.HeapSize > class file for org.apache.hadoop.hbase.io.HeapSize not found > > > Thanks!! > > On Thu, Oct 20, 2016 at 1:44 PM, Thomas Weise <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> This may also help: >> >> http://docs.datatorrent.com/troubleshooting/#hadoop-dependen >> cies-conflicts >> >> >> On Thu, Oct 20, 2016 at 11:39 AM, Thomas Weise <[email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> Please see the HBase dependency and its exclusions here: >>> >>> https://github.com/apache/apex-malhar/blob/master/contrib/pom.xml#L342 >>> >>> Thanks, >>> Thomas >>> >>> On Thu, Oct 20, 2016 at 9:07 AM, Jaspal Singh < >>> [email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>>> Team, >>>> >>>> While using the Hbase operator with Datatorrent application, we have >>>> added hbase client dependency in pom.xml. Do we need to exclude transitive >>>> hadoop dependencies using 'exclusion *' ? >>>> >>>> If we do that then HbaseConfiguration and Bytes methods are also >>>> getting excluded and giving error in the application. Is there a way to fix >>>> it ?? >>>> >>>> >>>> Thanks!! >>>> >>>> On Thu, Oct 20, 2016 at 1:38 AM, Tushar Gosavi <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >>>>> Hi Jaspal, >>>>> >>>>> You can pass the store name through property file, like >>>>> >>>>> <property> >>>>> <name>dt.operator.HbaseOperatorName.store.tableName</name> >>>>> <value>{name of the table}</value> >>>>> </property> >>>>> >>>>> In the code, you can set the table name and other properties in >>>>> constructor. >>>>> {code} >>>>> public static class Status2Hbase extends AbstractHBasePutOutputOperator >>>>> <Status> >>>>> { >>>>> >>>>> public Status2Hbase() >>>>> { >>>>> super(); >>>>> // store is initialized to HBaseStore >>>>> store.setTableName("nameofTable"); >>>>> } >>>>> >>>>> @Override >>>>> public Put operationPut(Status t) >>>>> { >>>>> Put put = new Put(ByteBuffer.allocate(8).put >>>>> Long(t.getCreatedAt().getTime()).array()); >>>>> put.add("cf".getBytes(), "text".getBytes(), t.getText().getBytes()); >>>>> put.add("cf".getBytes(), "userid".getBytes(), t.getText().getBytes()); >>>>> return put; >>>>> } >>>>> } >>>>> {code} >>>>> >>>>> - Tushar. >>>>> >>>>> >>>>> On Thu, Oct 20, 2016 at 11:59 AM, Jaspal Singh >>>>> <[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>> > Hi Thomas, Thanks for sharing this example code. >>>>> > Still I couldn't see where the hbase tablename is configured, it >>>>> says in >>>>> > description that it can be configured. >>>>> > >>>>> > Can you please highlight where it is specified ? >>>>> > >>>>> > Thanks!! >>>>> > >>>>> > >>>>> > On Wednesday, October 19, 2016, Thomas Weise <[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>>> >> >>>>> >> Here is an example that uses HBase that may be helpful: >>>>> >> >>>>> >> >>>>> >> https://github.com/apache/apex-malhar/blob/master/demos/twit >>>>> ter/src/main/java/com/datatorrent/demos/twitter/TwitterDumpH >>>>> BaseApplication.java >>>>> >> >>>>> >> Thomas >>>>> >> >>>>> >> On Wed, Oct 19, 2016 at 6:36 PM, Jaspal Singh < >>>>> [email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>> >> wrote: >>>>> >>> >>>>> >>> Where I need to set the table name. In property file or the >>>>> application >>>>> >>> code ? >>>>> >>> >>>>> >>> >>>>> >>> Thanks!! >>>>> >>> >>>>> >>> >>>>> >>> On Wednesday, October 19, 2016, Sanjay Pujare < >>>>> [email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>> >>> wrote: >>>>> >>>> >>>>> >>>> Take a look at >>>>> >>>> https://github.com/apache/apex-malhar/tree/master/contrib/sr >>>>> c/main/java/com/datatorrent/contrib/hbase >>>>> >>>> . There are multiple output operators there. >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> You specify the table name using HBaseStore.setTableName >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> From: "Bandaru, Srinivas" <[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>> >>>> Reply-To: <[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>> >>>> Date: Wednesday, October 19, 2016 at 3:09 PM >>>>> >>>> To: "[email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>" < >>>>> [email protected] >>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>>> >>>> Subject: Datatorrent operator for Hbase >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> Hi, >>>>> >>>> >>>>> >>>> I want to write the data from an operator to a hbase table. Which >>>>> >>>> operator I can use to write to Hbase table? >>>>> >>>> >>>>> >>>> Also how to specify the Hbase table name? >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> Thanks, >>>>> >>>> >>>>> >>>> Srinivas Bandaru >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> This e-mail, including attachments, may include confidential >>>>> and/or >>>>> >>>> proprietary information, and may be used only by the person or >>>>> entity >>>>> >>>> to which it is addressed. If the reader of this e-mail is not the >>>>> >>>> intended >>>>> >>>> recipient or his or her authorized agent, the reader is hereby >>>>> notified >>>>> >>>> that any dissemination, distribution or copying of this e-mail is >>>>> >>>> prohibited. If you have received this e-mail in error, please >>>>> notify the >>>>> >>>> sender by replying to this message and delete this e-mail >>>>> immediately. >>>>> >> >>>>> >> >>>>> > >>>>> >>>> >>>> >>> >> >
