Hi Roshan, I've been discussing about this with Hari Shreedharan in thread. Basically, I have already changed that point to allow HBASE sink and I've recompiled the flume-ng-embedded jar. Also, I've included flume-ng-auth library and executed the Embedded agent. At this moment, I've detected some error during starting Embedded Agent. It looks it can't access to HBase table list from Embedded agent code. Any idea to solve this?
Thanks for your help -----Original Message----- From: Roshan Naik [mailto:[email protected]] Sent: Wednesday, June 10, 2015 10:12 PM To: [email protected] Subject: Re: HBaseSink for Flume The Embedded agent actually has some code in it to limit itself only to a list of allowed sinks/sources/channels in EmbeddedAgentConfiguration.java Below is the list private static final String[] ALLOWED_SOURCES = { SOURCE_TYPE_EMBEDDED_ALIAS, SOURCE_TYPE_EMBEDDED, }; private static final String[] ALLOWED_CHANNELS = { CHANNEL_TYPE_MEMORY, CHANNEL_TYPE_FILE }; private static final String[] ALLOWED_SINKS = { SINK_TYPE_AVRO }; private static final String[] ALLOWED_SINK_PROCESSORS = { SINK_PROCESSOR_TYPE_DEFAULT, SINK_PROCESSOR_TYPE_FAILOVER, SINK_PROCESSOR_TYPE_LOAD_BALANCE }; -roshan On 6/10/15 12:37 PM, "Hari Shreedharan" <[email protected]> wrote: >Adding HBase and HDFS to embedded agent is non-trivial, especially with >changing versions and dependencies. It is unlikely your hack will >actually work without having everything hbase-client needs in its >classpath (take a look at how flume-ng script adds everything to its >own classpath). I would suggest running hbase classpath and adding >eveyrhing from that to your app¹s classpath and trying. > >Also, it looks like at some point the hbase sink is not started, not >sure why. Frankly, no one has really tested this, neither is it really >supported. > >Thanks, >Hari Shreedharan > > > > >> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel >><[email protected]> wrote: >> >> Hi, >> Yes, it looks it works including flume-ng-auth in the classpath. >>However, after configuring agent with properties it fails when tries >>to >>start: >> ... >> properties.put("hbaseSink.table", "Models"); >> ... >> agent.configure(properties); >> agent.start(); >> ... >> org.apache.flume.FlumeException: Could not load table, Models >>from Hbase >> SEVERE: ...Error while sending data to >>HBasejava.lang.IllegalStateException: Cannot put events unless started >> java.lang.IllegalStateException: Cannot put events unless started >> ... >> >> It looks it can't access to Hbase table defined in properties. Should >>I configure some property in the agent to access to HBase? >> >> Thanks again >> >> >> -----Original Message----- >> From: Hari Shreedharan [mailto:[email protected]] >> Sent: Saturday, June 06, 2015 12:08 AM >> To: [email protected] >> Subject: Re: HBaseSink for Flume >> >> No, you need to add flume-ng-auth module to your classpath. >> >> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel < >>[email protected]> wrote: >> >>> Hi Hari, >>> Thanks for the answer. I've changed it to "Hbase" and this error has >>> disappeared but it seems that there is a problem with sink factory >>> when looking for HBase sink: >>> >>> Jun 05, 2015 11:25:47 AM >>> com.sun.jersey.spi.container.ContainerResponse >>> mapMappableContainerException >>> SEVERE: The exception contained within MappableContainerException >>>could not be mapped to a response, re-throwing to the HTTP container >>> java.lang.NoClassDefFoundError: >>>org/apache/flume/auth/PrivilegedExecutor >>> at java.lang.Class.forName0(Native Method) >>> at java.lang.Class.forName(Class.java:191) >>> at >>> >>>org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory. >>>jav >>>a:67) >>> at >>> >>>org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java: >>>41) >>> at >>> >>>org.apache.flume.node.AbstractConfigurationProvider.loadSinks(Abstrac >>>tCo >>>nfigurationProvider.java:415) >>> at >>> >>>org.apache.flume.node.AbstractConfigurationProvider.getConfiguration( >>>A >>> bstractConfigurationProvider.java:103) >>> >>> Should I modify some class on org.apache.flume.sink package as well? >>> >>> Thanks >>> >>> >>> -----Original Message----- >>> From: Hari Shreedharan [mailto:[email protected]] >>> Sent: Thursday, June 04, 2015 12:45 AM >>> To: [email protected] >>> Subject: Re: HBaseSink for Flume >>> >>> Did you simply try ³hbase² rather than HBaseSink? >>> >>> Thanks, >>> Hari Shreedharan >>> >>> >>> >>> >>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel < >>> [email protected]> wrote: >>>> >>>> Hi devs, >>>> Sorry for asking in developer list but I¹d like to ask you about a >>>> very >>> technical issue and probably you know better than anyone about flume >>>code. >>> I want to include HBaseSink in EmbeddedAgent configuration but >>>currently seems not to be included, despite I¹ve changed the code and >>>compiled it >>> again: >>>> org.apache.flume.FlumeException: Component type of HBaseSink is not >>>> in >>> allowed types of [AVRO, HBASE, HDFS] >>>> >>>> Could you please let me know if there is somehow to add this >>> functionality to EmbeddedAgent class? >>>> >>>> Thanks in advance >>>> >>>> This e-mail and the documents attached are confidential and >>>> intended >>> solely for the addressee; it may also be privileged. If you receive >>>this e-mail in error, please notify the sender immediately and >>>destroy it. >>>> As its integrity cannot be secured on the Internet, the Atos group >>> liability cannot be triggered for the message content. Although the >>> sender endeavors to maintain a computer virus-free network, the >>> sender does not warrant that this transmission is virus-free and >>> will not be liable for any damages resulting from any virus transmitted. >>>> >>>> Este mensaje y los ficheros adjuntos pueden contener información >>> confidencial destinada solamente a la(s) persona(s) mencionadas >>> anteriormente y pueden estar protegidos por secreto profesional. >>>> Si usted recibe este correo electrónico por error, gracias por >>>> informar >>> inmediatamente al remitente y destruir el mensaje. >>>> Al no estar asegurada la integridad de este mensaje sobre la red, >>>> Atos >>> no se hace responsable por su contenido. Su contenido no constituye >>>ningún compromiso para el grupo Atos, salvo ratificación escrita por >>>ambas partes. >>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el >>> emisor no puede garantizar nada al respecto y no será responsable de >>> cualesquiera daños que puedan resultar de una transmisión de virus. >>> >>> This e-mail and the documents attached are confidential and intended >>>solely for the addressee; it may also be privileged. If you receive >>>this e-mail in error, please notify the sender immediately and >>>destroy it. >>> As its integrity cannot be secured on the Internet, the Atos group >>>liability cannot be triggered for the message content. Although the >>>sender endeavors to maintain a computer virus-free network, the >>>sender does not warrant that this transmission is virus-free and >>>will not be liable for any damages resulting from any virus transmitted. >>> >>> Este mensaje y los ficheros adjuntos pueden contener información >>>confidencial destinada solamente a la(s) persona(s) mencionadas >>>anteriormente y pueden estar protegidos por secreto profesional. >>> Si usted recibe este correo electrónico por error, gracias por >>>informar inmediatamente al remitente y destruir el mensaje. >>> Al no estar asegurada la integridad de este mensaje sobre la red, >>>Atos no se hace responsable por su contenido. Su contenido no >>>constituye ningún compromiso para el grupo Atos, salvo ratificación >>>escrita por ambas partes. >>> Aunque se esfuerza al máximo por mantener su red libre de virus, el >>>emisor no puede garantizar nada al respecto y no será responsable de >>>cualesquiera daños que puedan resultar de una transmisión de virus. >>> >> This e-mail and the documents attached are confidential and intended >>solely for the addressee; it may also be privileged. If you receive >>this e-mail in error, please notify the sender immediately and destroy it. >> As its integrity cannot be secured on the Internet, the Atos group >>liability cannot be triggered for the message content. Although the >>sender endeavors to maintain a computer virus-free network, the sender >>does not warrant that this transmission is virus-free and will not be >>liable for any damages resulting from any virus transmitted. >> >> Este mensaje y los ficheros adjuntos pueden contener información >>confidencial destinada solamente a la(s) persona(s) mencionadas >>anteriormente y pueden estar protegidos por secreto profesional. >> Si usted recibe este correo electrónico por error, gracias por >>informar inmediatamente al remitente y destruir el mensaje. >> Al no estar asegurada la integridad de este mensaje sobre la red, >>Atos no se hace responsable por su contenido. Su contenido no >>constituye ningún compromiso para el grupo Atos, salvo ratificación >>escrita por ambas partes. >> Aunque se esfuerza al máximo por mantener su red libre de virus, el >>emisor no puede garantizar nada al respecto y no será responsable de >>cualesquiera daños que puedan resultar de una transmisión de virus. > This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional. Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje. Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes. Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
