Re: [graylog2] confused how extractor fields work

[Jason Haar]

On 23/11/15 10:58, Joi Owen wrote:
>
> I think your best bet is to split that one big text section you're
> currently using into separate extracted fields.  Use the original
> regexp as the control test, the one that decides if a particular log
> line is going to be further inspected for the matching field.
>

That really doesn't scale. eg I'm inserting our proxy server logs into
graylog, and I want to search to find all urls that end in "/73.exe" -
eg "/one/73.exe" and "/two/73.exe?id=112". Separating all urls into
single-word fields would add nearly infinite numbers of fields - not
really an option

As you mentioned earlier, the documentation says "Also note that
message, full_message, and source are the only fields that can be
searched via wildcard by default". So that implies I should be able to
make other fields act like the message field

So how can I make my "url" field properly searchable? (ie to handle
wildcards)

Thanks

-- 
Cheers

Jason Haar
Corporate Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/56634E6D.3060404%40trimble.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] confused how extractor fields work

[Joi Owen]

I suspect it has more to do with how graylog tells elasticsearch to index
those fields.  Timestamp searches are always a bit hairy due to how
timestamps are stored, how locale/TZ settings affect the displays, etc.
I've never had the need to include a timestamp search that couldn't be
easily produced by dragging the selection bars within the graphical display
of the data, anyway.

I think your best bet is to split that one big text section you're
currently using into separate extracted fields.  Use the original regexp as
the control test, the one that decides if a particular log line is going to
be further inspected for the matching field.



On Sun, Nov 22, 2015 at 2:32 PM, Jason Haar  wrote:

> On 23/11/15 04:52, Joi Owen wrote:
>
>
> Also note that message, full_message, and source are the only fields that
> can be searched via wildcard by default."
>
> I find that if I extract fields that are single words, I can search for
> them by exact match, and oddly, I am able to do a field match with a
> trailing *, but not a leading one, and not a subset of the word.  Ie I can
> do exact match, and 'starts with' but nothing else on my extracted field.
>
>
> Yeah that's what I'm thinking too. I'd love to hear precisely what are the
> search options. To formalize what you're saying, it looks to me like
> graylog can only search under the following constraints:
>
> 1. "message" field is "special" and supports wildcards and pseudo-regex
> 2. timestamp is handled outside the search bar
> 3. any other field must be a precise match  - although you say you can
> wildcard the end of it (I haven't tested myself because frankly it's pretty
> useless to me)
>
> I'm guessing this is really an elasticsearch limitation? Is there any way
> for graylog to tell elasticsearch to treat certain other fields the same
> way as the "message" field? (or at least to match on substrings). If so, it
> could be exposed via the extractor wizard - eg add "allow substring
> matches" as a drop-down option?
>
> --
> Cheers
>
> Jason Haar
> Corporate Information Security Manager, Trimble Navigation Ltd.
> Phone: +1 408 481 8171
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/56522663.8030709%40trimble.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 

No matter what we think of Linux versus FreeBSD, etc., the one thing I
really like about Linux is that it has Microsoft worried. Anything
that kicks a monopoly in the pants has got to be good for something.
- Chris Johnson

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAL5rfGUjjJd4sGWGb6p7PtM9wy_F%2BN6wPCtW1HuO4d6iy73e2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] confused how extractor fields work

[Jason Haar]

On 23/11/15 04:52, Joi Owen wrote:
>
> Also note that message, full_message, and source are the only fields
> that can be searched via wildcard by default."
>
> I find that if I extract fields that are single words, I can search
> for them by exact match, and oddly, I am able to do a field match with
> a trailing *, but not a leading one, and not a subset of the word.  Ie
> I can do exact match, and 'starts with' but nothing else on my
> extracted field.

Yeah that's what I'm thinking too. I'd love to hear precisely what are
the search options. To formalize what you're saying, it looks to me like
graylog can only search under the following constraints:

1. "message" field is "special" and supports wildcards and pseudo-regex
2. timestamp is handled outside the search bar
3. any other field must be a precise match  - although you say you can
wildcard the end of it (I haven't tested myself because frankly it's
pretty useless to me)

I'm guessing this is really an elasticsearch limitation? Is there any
way for graylog to tell elasticsearch to treat certain other fields the
same way as the "message" field? (or at least to match on substrings).
If so, it could be exposed via the extractor wizard - eg add "allow
substring matches" as a drop-down option?

-- 
Cheers

Jason Haar
Corporate Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/56522663.8030709%40trimble.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] confused how extractor fields work

[Joi Owen]

I find this bit in the help text for the search bar:

"Note that leading wildcards are disabled to avoid excessive memory
consumption! You can enable them in your graylog-server.conf:
allow_leading_wildcard_searches = true

Also note that message, full_message, and source are the only fields that
can be searched via wildcard by default."

I find that if I extract fields that are single words, I can search for
them by exact match, and oddly, I am able to do a field match with a
trailing *, but not a leading one, and not a subset of the word.  Ie I can
do exact match, and 'starts with' but nothing else on my extracted field.

I've noticed that regular expressions like those you use in the extractors
don't work in the search bar itself, the syntax there is much more basic
and limited.

I don't use extractors to pull out entire phrases, I create individual
fields for the pieces of the phrase I care about and leave it at that.  I
might use the phrase format as the control expression, though.



On Sun, Nov 22, 2015 at 2:48 AM, Jason Haar  wrote:

> Hi there
>
> I created an extractor to extract part of a record and associate it with
> a new fieldname. That works fine - expanding affected records inside
> graylog-web shows the new field shows up with the correct sub-section of
> the original message in it.
>
> However, if I search for a word inside that matched field, I get no
> hits. When I do the same search using the default "message" field - I
> get the hits expected.
>
> So what I have I missed? When you create new fields using extractors,
> how do you ensure they are searchable using (I guess) substring matches?
> It was a "word" that showed up in the middle of the "sentence" that was
> associated with the new fieldname - ie pretty simple. And yet I couldn't
> search for a word? The converter on the extractor was the default
> "Numeric" - should it be something else?
>
> This is graylog-1.2.2
>
> Thanks
>
> --
> Cheers
>
> Jason Haar
> Corporate Information Security Manager, Trimble Navigation Ltd.
> Phone: +1 408 481 8171
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/5651814D.703%40trimble.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 

No matter what we think of Linux versus FreeBSD, etc., the one thing I
really like about Linux is that it has Microsoft worried. Anything
that kicks a monopoly in the pants has got to be good for something.
- Chris Johnson

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAL5rfGWUM3e%2BvxEOsYn%2BSOCb%2BqeWO%2BoHCYYGJFByGP4F8EudHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] confused how extractor fields work

[Jason Haar]

Hi there

I created an extractor to extract part of a record and associate it with
a new fieldname. That works fine - expanding affected records inside
graylog-web shows the new field shows up with the correct sub-section of
the original message in it.

However, if I search for a word inside that matched field, I get no
hits. When I do the same search using the default "message" field - I
get the hits expected.

So what I have I missed? When you create new fields using extractors,
how do you ensure they are searchable using (I guess) substring matches?
It was a "word" that showed up in the middle of the "sentence" that was
associated with the new fieldname - ie pretty simple. And yet I couldn't
search for a word? The converter on the extractor was the default
"Numeric" - should it be something else?

This is graylog-1.2.2

Thanks

-- 
Cheers

Jason Haar
Corporate Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5651814D.703%40trimble.com.
For more options, visit https://groups.google.com/d/optout.