Solrs stats functions are great when analyzing logs if they are pre processed.
> On Feb 21, 2022, at 4:26 PM, Joel Bernstein wrote:
>
> We use Solr for logs analytics. This is a lot more power in Solr's math
> expressions than in Elastic's aggregations and Solr also has new root cause
>
We use Solr for logs analytics. This is a lot more power in Solr's math
expressions than in Elastic's aggregations and Solr also has new root cause
analysis and event correlation query. Here are some links:
https://solr.apache.org/guide/8_11/math-expressions.html
Thank you for the reply, unfortunately, I had looked at the documentation and
found it lacking since there are no examples...
When I look at filterCache it makes sense on what it does and to a certain
extent how it does it. Same goes with queryResultCache.
While both of these provide a list
No, we use a more general approach to collect all kinds of logfiles
TailFile -> ExtractGrok -> JoltTransformJSON (make some transformation) ->
PutSolrRecord
And we dont use Nifi directly, we use Minifi running on each machine. So we can
collect all kinds of logfiles from all kinds of
We lost or zookeeper data about a week ago due to /queue-work being too
large or corrupt. We were unable to access/remove the data in the zk folder
/queue-work.
To get around that we created a new ZK instance and repopulated it with the
custom config used by the application and repopulated our
> yes solr is suitable for this. We aggregate various logfiles from many
> hosts with minifi and send them line by line to solr. Ingestion and
> indexing is fine and you can query the logfiles just moments after
> ingestion.
Thank you for the idea. I have no experience with Apache NiFi at the
While this is still a bit “roll your own”, you may be interested in
https://solr.apache.org/guide/8_11/logs.html
> On Feb 21, 2022, at 3:54 AM, Z0ltrix wrote:
>
> Hi Sam Lee,
>
> yes solr is suitable for this. We aggregate various logfiles from many hosts
> with minifi and send them line
Hi Sam Lee,
yes solr is suitable for this. We aggregate various logfiles from many hosts
with minifi and send them line by line to solr. Ingestion and indexing is fine
and you can query the logfiles just moments after ingestion.
Regards,
Christian
--- Original Message ---
Sam Lee
Is Apache Solr suitable for use as a log aggregation and near-real-time
log querying tool?
I am thinking of using Apache Solr to store syslog messages as they
arrive. For example, by using rsyslog with its "pipe output module"
(ompipe) that pipes logs into a Python/Ruby/Shell script that then
