The Embedded agent actually has some code in it to limit itself only to a
list of allowed sinks/sources/channels in EmbeddedAgentConfiguration.java
Below is the list
private static final String[] ALLOWED_SOURCES = {
SOURCE_TYPE_EMBEDDED_ALIAS,
SOURCE_TYPE_EMBEDDED,
};
private static final String[] ALLOWED_CHANNELS = {
CHANNEL_TYPE_MEMORY,
CHANNEL_TYPE_FILE
};
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO
};
private static final String[] ALLOWED_SINK_PROCESSORS = {
SINK_PROCESSOR_TYPE_DEFAULT,
SINK_PROCESSOR_TYPE_FAILOVER,
SINK_PROCESSOR_TYPE_LOAD_BALANCE
};
-roshan
On 6/10/15 12:37 PM, "Hari Shreedharan" <[email protected]> wrote:
>Adding HBase and HDFS to embedded agent is non-trivial, especially with
>changing versions and dependencies. It is unlikely your hack will
>actually work without having everything hbase-client needs in its
>classpath (take a look at how flume-ng script adds everything to its own
>classpath). I would suggest running hbase classpath and adding eveyrhing
>from that to your app¹s classpath and trying.
>
>Also, it looks like at some point the hbase sink is not started, not sure
>why. Frankly, no one has really tested this, neither is it really
>supported.
>
>Thanks,
>Hari Shreedharan
>
>
>
>
>> On Jun 10, 2015, at 3:13 AM, Tinte garcia, Miguel Angel
>><[email protected]> wrote:
>>
>> Hi,
>> Yes, it looks it works including flume-ng-auth in the classpath.
>>However, after configuring agent with properties it fails when tries to
>>start:
>> ...
>> properties.put("hbaseSink.table", "Models");
>> ...
>> agent.configure(properties);
>> agent.start();
>> ...
>> org.apache.flume.FlumeException: Could not load table, Models
>>from Hbase
>> SEVERE: ...Error while sending data to
>>HBasejava.lang.IllegalStateException: Cannot put events unless started
>> java.lang.IllegalStateException: Cannot put events unless started
>> ...
>>
>> It looks it can't access to Hbase table defined in properties. Should I
>>configure some property in the agent to access to HBase?
>>
>> Thanks again
>>
>>
>> -----Original Message-----
>> From: Hari Shreedharan [mailto:[email protected]]
>> Sent: Saturday, June 06, 2015 12:08 AM
>> To: [email protected]
>> Subject: Re: HBaseSink for Flume
>>
>> No, you need to add flume-ng-auth module to your classpath.
>>
>> On Fri, Jun 5, 2015 at 2:38 AM, Tinte garcia, Miguel Angel <
>>[email protected]> wrote:
>>
>>> Hi Hari,
>>> Thanks for the answer. I've changed it to "Hbase" and this error has
>>> disappeared but it seems that there is a problem with sink factory
>>> when looking for HBase sink:
>>>
>>> Jun 05, 2015 11:25:47 AM
>>> com.sun.jersey.spi.container.ContainerResponse
>>> mapMappableContainerException
>>> SEVERE: The exception contained within MappableContainerException
>>> could not be mapped to a response, re-throwing to the HTTP container
>>> java.lang.NoClassDefFoundError:
>>>org/apache/flume/auth/PrivilegedExecutor
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:191)
>>> at
>>>
>>>org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.jav
>>>a:67)
>>> at
>>>
>>>org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:
>>>41)
>>> at
>>>
>>>org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractCo
>>>nfigurationProvider.java:415)
>>> at
>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(A
>>> bstractConfigurationProvider.java:103)
>>>
>>> Should I modify some class on org.apache.flume.sink package as well?
>>>
>>> Thanks
>>>
>>>
>>> -----Original Message-----
>>> From: Hari Shreedharan [mailto:[email protected]]
>>> Sent: Thursday, June 04, 2015 12:45 AM
>>> To: [email protected]
>>> Subject: Re: HBaseSink for Flume
>>>
>>> Did you simply try ³hbase² rather than HBaseSink?
>>>
>>> Thanks,
>>> Hari Shreedharan
>>>
>>>
>>>
>>>
>>>> On Jun 3, 2015, at 9:18 AM, Tinte garcia, Miguel Angel <
>>> [email protected]> wrote:
>>>>
>>>> Hi devs,
>>>> Sorry for asking in developer list but I¹d like to ask you about a
>>>> very
>>> technical issue and probably you know better than anyone about flume
>>>code.
>>> I want to include HBaseSink in EmbeddedAgent configuration but
>>> currently seems not to be included, despite I¹ve changed the code and
>>> compiled it
>>> again:
>>>> org.apache.flume.FlumeException: Component type of HBaseSink is not
>>>> in
>>> allowed types of [AVRO, HBASE, HDFS]
>>>>
>>>> Could you please let me know if there is somehow to add this
>>> functionality to EmbeddedAgent class?
>>>>
>>>> Thanks in advance
>>>>
>>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy
>>>it.
>>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the sender
>>> does not warrant that this transmission is virus-free and will not be
>>> liable for any damages resulting from any virus transmitted.
>>>>
>>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>>> Si usted recibe este correo electrónico por error, gracias por
>>>> informar
>>> inmediatamente al remitente y destruir el mensaje.
>>>> Al no estar asegurada la integridad de este mensaje sobre la red,
>>>> Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>>ambas partes.
>>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive
>>> this e-mail in error, please notify the sender immediately and destroy
>>>it.
>>> As its integrity cannot be secured on the Internet, the Atos group
>>> liability cannot be triggered for the message content. Although the
>>> sender endeavors to maintain a computer virus-free network, the sender
>>> does not warrant that this transmission is virus-free and will not be
>>> liable for any damages resulting from any virus transmitted.
>>>
>>> Este mensaje y los ficheros adjuntos pueden contener información
>>> confidencial destinada solamente a la(s) persona(s) mencionadas
>>> anteriormente y pueden estar protegidos por secreto profesional.
>>> Si usted recibe este correo electrónico por error, gracias por
>>> informar inmediatamente al remitente y destruir el mensaje.
>>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos
>>> no se hace responsable por su contenido. Su contenido no constituye
>>> ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>>ambas partes.
>>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>> emisor no puede garantizar nada al respecto y no será responsable de
>>> cualesquiera daños que puedan resultar de una transmisión de virus.
>>>
>> This e-mail and the documents attached are confidential and intended
>>solely for the addressee; it may also be privileged. If you receive this
>>e-mail in error, please notify the sender immediately and destroy it.
>> As its integrity cannot be secured on the Internet, the Atos group
>>liability cannot be triggered for the message content. Although the
>>sender endeavors to maintain a computer virus-free network, the sender
>>does not warrant that this transmission is virus-free and will not be
>>liable for any damages resulting from any virus transmitted.
>>
>> Este mensaje y los ficheros adjuntos pueden contener información
>>confidencial destinada solamente a la(s) persona(s) mencionadas
>>anteriormente y pueden estar protegidos por secreto profesional.
>> Si usted recibe este correo electrónico por error, gracias por informar
>>inmediatamente al remitente y destruir el mensaje.
>> Al no estar asegurada la integridad de este mensaje sobre la red, Atos
>>no se hace responsable por su contenido. Su contenido no constituye
>>ningún compromiso para el grupo Atos, salvo ratificación escrita por
>>ambas partes.
>> Aunque se esfuerza al máximo por mantener su red libre de virus, el
>>emisor no puede garantizar nada al respecto y no será responsable de
>>cualesquiera daños que puedan resultar de una transmisión de virus.
>
