You can also refer :
http://eagle.apache.org/docs/import-hdfs-auditLog.html
<http://eagle.apache.org/docs/import-hdfs-auditLog.html>
Try having the log stash conf filter and output similar to this:
filter {
grok {
match => ["message",
"ugi=(?<user>([\w\d\-]+))@|ugi=(?<user>([\w\d\-]+))/[\w\d\-.]+@|ugi=(?<user>([\w\d.\-_]+))[\s(]+"]
}
}
output {
stdout {
codec => rubydebug
}
kafka {
codec => plain {
format => "%{message}"
}
bootstrap_servers => "127.0.0.1:6667"
topic_id => "TestTopic"
retry_backoff_ms => 100
send_buffer_bytes => 102400
}
}
-Sudha Jenslin
> On Mar 21, 2017, at 7:53 PM, Jean Rossier <[email protected]> wrote:
>
> Hello,
>
> I'm trying to setup the HDFS data activity monitoring stream (as explained
> here http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html
> <http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html>), but I
> don't see any logs flowing into my Kafka topic.
>
> Is there anything to do that is not on the quick start page (mentioned above)
> ?
>
> One thing that surprised me, is that I don't have any log4jkafka folder in
> /usr/hdp/current/eagle/lib/ (I'm using eagle 0.5.0-SNAPSHOT, from current
> master branch). I'm supposed to add the libs from this folder to the
> HADOOP_CLASSPATH env variable. I moved them by hand in the proper folder,
> but, still, I don't see any logs flowing into my Kafka topic.
>
> I don't see any error regarding kafka in the HDFS audit logs. There are,
> though, some solr exceptions but I guess it shouldn't impact the stream I'm
> interested in.
>
> Thanks
> Jean
>
