Team, While building the datatorrent application with Kafka as source, we are getting the below errors. Any inputs ??
NOTE: We have excluded the hadoop dependencies from the hbase-client in pom.xml. [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[13,23] package org.apache.http does not exist [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[14,30] package org.apache.http.client does not exist [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[15,38] package org.apache.http.client.methods does not exist [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[16,38] package org.apache.http.client.methods does not exist [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[17,30] package org.apache.http.entity does not exist [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[18,35] package org.apache.http.impl.client does not exist [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[87,13] cannot find symbol symbol: class HttpClient location: class com.example.datatorrent.HbaseTableUpdate<T> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[87,41] cannot find symbol symbol: class DefaultHttpClient location: class com.example.datatorrent.HbaseTableUpdate<T> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[88,13] cannot find symbol symbol: class HttpPut location: class com.example.datatorrent.HbaseTableUpdate<T> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[88,35] cannot find symbol symbol: class HttpPut location: class com.example.datatorrent.HbaseTableUpdate<T> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[89,13] cannot find symbol symbol: class StringEntity location: class com.example.datatorrent.HbaseTableUpdate<T> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[89,39] cannot find symbol symbol: class StringEntity location: class com.example.datatorrent.HbaseTableUpdate<T> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-kafka/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[92,13] cannot find symbol symbol: class HttpResponse location: class com.example.datatorrent.HbaseTableUpdate<T> Thanks!! On Thu, Oct 27, 2016 at 9:22 AM, Jaspal Singh <[email protected]> wrote: > Team, > > Is there any exactly once Hbase put operator ?? > > Also can we define two tables using the set table property for Hbase store > ?? > > Thanks!! > > > On Thursday, October 20, 2016, Jaspal Singh <[email protected]> > wrote: > >> Thomas, >> >> When we try excluding the transitive dependencies using 'exclusion *', >> getting the below error while running maven build: >> >> [ERROR] COMPILATION ERROR : >> [INFO] ------------------------------------------------------------- >> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-mapr-streams/s >> rc/main/java/com/example/datatorrent/HbaseTableUpdate.java:[51,18] >> cannot access org.apache.hadoop.hbase.CellScannable >> class file for org.apache.hadoop.hbase.CellScannable not found >> [ERROR] /C:/Work/Datatorrent-projects/bdp-datatorrent-mapr-streams/s >> rc/main/java/com/example/datatorrent/HbaseTableUpdate.java:[105,10] >> cannot access org.apache.hadoop.hbase.io.HeapSize >> class file for org.apache.hadoop.hbase.io.HeapSize not found >> >> >> Thanks!! >> >> On Thu, Oct 20, 2016 at 1:44 PM, Thomas Weise <[email protected]> wrote: >> >>> This may also help: >>> >>> http://docs.datatorrent.com/troubleshooting/#hadoop-dependen >>> cies-conflicts >>> >>> >>> On Thu, Oct 20, 2016 at 11:39 AM, Thomas Weise <[email protected]> wrote: >>> >>>> Please see the HBase dependency and its exclusions here: >>>> >>>> https://github.com/apache/apex-malhar/blob/master/contrib/pom.xml#L342 >>>> >>>> Thanks, >>>> Thomas >>>> >>>> On Thu, Oct 20, 2016 at 9:07 AM, Jaspal Singh < >>>> [email protected]> wrote: >>>> >>>>> Team, >>>>> >>>>> While using the Hbase operator with Datatorrent application, we have >>>>> added hbase client dependency in pom.xml. Do we need to exclude transitive >>>>> hadoop dependencies using 'exclusion *' ? >>>>> >>>>> If we do that then HbaseConfiguration and Bytes methods are also >>>>> getting excluded and giving error in the application. Is there a way to >>>>> fix >>>>> it ?? >>>>> >>>>> >>>>> Thanks!! >>>>> >>>>> On Thu, Oct 20, 2016 at 1:38 AM, Tushar Gosavi <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi Jaspal, >>>>>> >>>>>> You can pass the store name through property file, like >>>>>> >>>>>> <property> >>>>>> <name>dt.operator.HbaseOperatorName.store.tableName</name> >>>>>> <value>{name of the table}</value> >>>>>> </property> >>>>>> >>>>>> In the code, you can set the table name and other properties in >>>>>> constructor. >>>>>> {code} >>>>>> public static class Status2Hbase extends >>>>>> AbstractHBasePutOutputOperator<Status> >>>>>> { >>>>>> >>>>>> public Status2Hbase() >>>>>> { >>>>>> super(); >>>>>> // store is initialized to HBaseStore >>>>>> store.setTableName("nameofTable"); >>>>>> } >>>>>> >>>>>> @Override >>>>>> public Put operationPut(Status t) >>>>>> { >>>>>> Put put = new Put(ByteBuffer.allocate(8).put >>>>>> Long(t.getCreatedAt().getTime()).array()); >>>>>> put.add("cf".getBytes(), "text".getBytes(), t.getText().getBytes()); >>>>>> put.add("cf".getBytes(), "userid".getBytes(), t.getText().getBytes()); >>>>>> return put; >>>>>> } >>>>>> } >>>>>> {code} >>>>>> >>>>>> - Tushar. >>>>>> >>>>>> >>>>>> On Thu, Oct 20, 2016 at 11:59 AM, Jaspal Singh >>>>>> <[email protected]> wrote: >>>>>> > Hi Thomas, Thanks for sharing this example code. >>>>>> > Still I couldn't see where the hbase tablename is configured, it >>>>>> says in >>>>>> > description that it can be configured. >>>>>> > >>>>>> > Can you please highlight where it is specified ? >>>>>> > >>>>>> > Thanks!! >>>>>> > >>>>>> > >>>>>> > On Wednesday, October 19, 2016, Thomas Weise <[email protected]> >>>>>> wrote: >>>>>> >> >>>>>> >> Here is an example that uses HBase that may be helpful: >>>>>> >> >>>>>> >> >>>>>> >> https://github.com/apache/apex-malhar/blob/master/demos/twit >>>>>> ter/src/main/java/com/datatorrent/demos/twitter/TwitterDumpH >>>>>> BaseApplication.java >>>>>> >> >>>>>> >> Thomas >>>>>> >> >>>>>> >> On Wed, Oct 19, 2016 at 6:36 PM, Jaspal Singh < >>>>>> [email protected]> >>>>>> >> wrote: >>>>>> >>> >>>>>> >>> Where I need to set the table name. In property file or the >>>>>> application >>>>>> >>> code ? >>>>>> >>> >>>>>> >>> >>>>>> >>> Thanks!! >>>>>> >>> >>>>>> >>> >>>>>> >>> On Wednesday, October 19, 2016, Sanjay Pujare < >>>>>> [email protected]> >>>>>> >>> wrote: >>>>>> >>>> >>>>>> >>>> Take a look at >>>>>> >>>> https://github.com/apache/apex-malhar/tree/master/contrib/sr >>>>>> c/main/java/com/datatorrent/contrib/hbase >>>>>> >>>> . There are multiple output operators there. >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> You specify the table name using HBaseStore.setTableName >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> From: "Bandaru, Srinivas" <[email protected]> >>>>>> >>>> Reply-To: <[email protected]> >>>>>> >>>> Date: Wednesday, October 19, 2016 at 3:09 PM >>>>>> >>>> To: "[email protected]" <[email protected]> >>>>>> >>>> Subject: Datatorrent operator for Hbase >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> Hi, >>>>>> >>>> >>>>>> >>>> I want to write the data from an operator to a hbase table. >>>>>> Which >>>>>> >>>> operator I can use to write to Hbase table? >>>>>> >>>> >>>>>> >>>> Also how to specify the Hbase table name? >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> Thanks, >>>>>> >>>> >>>>>> >>>> Srinivas Bandaru >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> This e-mail, including attachments, may include confidential >>>>>> and/or >>>>>> >>>> proprietary information, and may be used only by the person or >>>>>> entity >>>>>> >>>> to which it is addressed. If the reader of this e-mail is not the >>>>>> >>>> intended >>>>>> >>>> recipient or his or her authorized agent, the reader is hereby >>>>>> notified >>>>>> >>>> that any dissemination, distribution or copying of this e-mail is >>>>>> >>>> prohibited. If you have received this e-mail in error, please >>>>>> notify the >>>>>> >>>> sender by replying to this message and delete this e-mail >>>>>> immediately. >>>>>> >> >>>>>> >> >>>>>> > >>>>>> >>>>> >>>>> >>>> >>> >>
