[graylog2] Re: Graylog graylog-2.1.0-rc.1.tgz Web Interface not loading

[123Dev]

I should also point out that we're not running behind a reverse proxy and 
this is a single node server with ES and MongoDB running on the same server.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/45474dee-b3d5-4f7c-9505-8b90e8d9489e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog graylog-2.1.0-rc.1.tgz Web Interface not loading

[123Dev]

Just upgraded to graylog-2.1.0-rc.1 using graylog-2.1.0-rc.1.tgz 

Web Interface does not come up on http://:9000
however API endpoint http://http://10.20.2.75:9000/api/


Graylog log doesn't show any errors

Any idea what it might be.
I'll turn off some of the plugins that were enabled in 2.0.2 and see if any 
of those makes a difference.
Anything persistent in MongDB that could cause this?

Thanks


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/de5d1ea5-6131-40cd-86e0-e55ca1083323%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: [ANNOUNCE] Graylog v2.1.0-RC.1 has been released

[123Dev]

Thanks Jochen,



On Friday, August 26, 2016 at 3:19:54 AM UTC-4, Jochen Schalanda wrote:
>
> Hi,
>
> On Thursday, 25 August 2016 23:33:54 UTC+2, 123Dev wrote:
>>
>> Or is it simply following these steps.
>>
>> http://docs.graylog.org/en/latest/pages/configuration/graylog_ctl.html#upgrade-graylog
>> Which were used to upgrade from 2.0 to 2.0.3
>>
>
> Yes, that should still work but make sure to read through 
> https://github.com/Graylog2/graylog2-server/blob/2.1.0-rc.1/UPGRADING.rst 
> before upgrading.
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a75b777d-5b7e-48e8-a6c3-1e64735c1f19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Autmatically parsed fields in Syslog TCP/UDP input

[Markus Fischbacher]

Anton know a way to extract syslog Levels. That doesn’t comes in the
message itself. Level and facility should be udp section?

I already had some good amount of googling. Found some good sources for the
extractors. Also i love the marketplace. One of the best of an open source
project.

Well if i have some spare time this Fall i try to submit an pushen request
or at least an plugin.

Jan Doberstein  schrieb am Fr., 26. Aug. 2016, 12:34:

> Hej Markus,
>
>
> I filed an enhancement in github (
> https://github.com/Graylog2/graylog2-server/issues/2739 ) but that was
> closed quickly with the "tip" to just use Raw Text Input - which isn't a
> solution because that Input is lacking Syslog fields i need ( level,
> facility, ... ).
>
> take a look what other already do:
> https://marketplace.graylog.org/addons?search=vmware
>
> personal i would use the „raw input“ and extract the fields i like to see
> with the give groks in the marketplace - or take a google / github search.
> I was able to identify different resources that can help you with this.
>
>
> /jd
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/graylog2/jj6mdnQ4B40/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/etPan.57c01b36.bb31434.ae4f%40jalogisch.de
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
-- 

BG
Markus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CALei8VQJ8WV6zQWc8XYjodFTyqjJaJ%2BxLO1jDoefe5AobnDeig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog Cluster - Adding a Second Node

[Dustin Tennill]

All,

We currently have graylog deployed like this: 

Graylog ServerA (2.0.3) - 40G Ram, 20G Heap, runs MongoDB, currently takes 
ALL incoming logs. 
Graylog ServerB (2.0.3) - 40G Ram, 20G Heap, currently takes NO incoming 
logs. 
Three ElasticSearch Nodes (2.3.5) - 64G Ram each, ~30G Heap. 
Statistics: 3500 msgs per second avg
We are not using a load balancer for log data yet, so all traffic goes to 
ServerA right now. 

We have moved about 60% of our logging traffic into it, and expect when 
finished to be at about 16k msg per second. While working through the 
project, we have added various streams and extractors as required. The goal 
was to keep these as few as possible, but the huge variety of logs input 
and formats has led to a fair number. 

We wanted to add a second node to graylog (not elasticsearch, but actual 
graylog) to prepare to spread out the processing load. The server will be 
called "ServerB" as listed above. We created the second node, copied 
relevant config, pointed it at our DBs (elasticsearch and mongo) and 
started things up. The node starts just fine, and appears to be basically 
healthy. 

Health looking things: 
1. The In/Out counter at the top of the screen is running, and showing 
numbers we expect (~3500 per second) - this makes us think our ES 
connection is fine. 
2. Users can login with ldap creds - this makes us think MongoDB 
connections are fine. 
3. Streams - the counters show correctly as streams get messages. 

Now for the issue(s):
1. We see only incoming log message from a single source when searching the 
last five minutes. It is always the same source. This happens even we KNOW 
there are other log data from past five minutes. If we change to the past 
hour, all logs are there and appear correct. If we search past 15 minutes, 
we see all log data. Sometimes we log into the second node and can only see 
messages from this single source. 
2. Streams - while the counts are there and appear correct, actually 
clicking into a stream and searching doesn't show any messages. Again, if 
you search past the 15 minute mark all messages are visible. 

Is this normal? I couldn't find a guide or set of specific instructions for 
what to do on the second node. It all seemed obvious, but I am wondering 
what we missed. 

Any pointers? 

Thanks !!!

Dustin Tennill








-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/31271ebf-c1d4-4459-9f20-8aa61ca48103%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog not writing to elasticsearch after out of disk space, ES green but...

[Obie]

Awesome, thanks!

That worked for me.One command difference is I had to drop the * like:
rm -r /var/opt/graylog/data/journal/

Thanks!!!

On Friday, August 26, 2016 at 10:26:26 AM UTC-4, juli...@gmail.com wrote:
>
> This is what I figured out to have it back on track:
>
> graylog-ctl stop
> rm -r /var/opt/graylog/data/journal/*
> graylog-ctl start
>
> Then cycle deflector (very important) and you should have it back on 
> track. Unfortunately, you will lose all the journal messages.
> I tried deleting only parts of the journal (those corrupted) but that 
> didn't work to reset the counter properly. Only a complete deletion worked 
> for me. 
>
> I hope it helps :)
>
> On Friday, 26 August 2016 10:03:12 UTC-4, Obie wrote:
>>
>> Some commands or a link to a doc would be helpful here.
>>
>> On Thursday, August 4, 2016 at 12:53:46 PM UTC-4, juli...@gmail.com 
>> wrote:
>>>
>>> Ok so what are those commands?? Can you provide a link or something? 
>>> Googling what you mention isn't very explicit and results aren't helpful. 
>>> Thx
>>>
>>> On Tuesday, 5 April 2016 04:02:08 UTC-4, Jochen Schalanda wrote:

 Hi,

 it looks like your journal is corrupted. You can either try to recover 
 and repair it (see the journal-related commands in Graylog) or simply 
 remove the journal files from disk.

 Cheers,
 Jochen

 On Monday, 4 April 2016 22:56:47 UTC+2, kluch wrote:
>
> After "out of disk space" I removed journals, old indices, cycled 
> deflector and then even deleted all indices from ES but it not helped at 
> all. All graylog nodes restarted and still nothing.
> I dont know from where graylog reads/displays negative (with minus) 
> value of unprocessed messages.  On web it looks like this:
> Processing *1* incoming and *0* outgoing msg/s. *-1,195,856,763 
> unprocessed messages* are currently in the journal, in 1 segments. *1 
> messages* have been appended to, and *0 messages* have been read from 
> the journal in the last second.
> Moreover graylog does not write to elasticsearch so only unprocessed 
> messages counter is changing. Time is in sync with ntp. Version 1.3.4 
> from OVA.
> Any help?
>


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/fbf37c93-9a08-48e7-b68e-98a96ecc33a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog not writing to elasticsearch after out of disk space, ES green but...

[julioqc47]

This is what I figured out to have it back on track:

graylog-ctl stop
rm -r /var/opt/graylog/data/journal/*
graylog-ctl start

Then cycle deflector (very important) and you should have it back on track. 
Unfortunately, you will lose all the journal messages.
I tried deleting only parts of the journal (those corrupted) but that 
didn't work to reset the counter properly. Only a complete deletion worked 
for me. 

I hope it helps :)

On Friday, 26 August 2016 10:03:12 UTC-4, Obie wrote:
>
> Some commands or a link to a doc would be helpful here.
>
> On Thursday, August 4, 2016 at 12:53:46 PM UTC-4, juli...@gmail.com wrote:
>>
>> Ok so what are those commands?? Can you provide a link or something? 
>> Googling what you mention isn't very explicit and results aren't helpful. 
>> Thx
>>
>> On Tuesday, 5 April 2016 04:02:08 UTC-4, Jochen Schalanda wrote:
>>>
>>> Hi,
>>>
>>> it looks like your journal is corrupted. You can either try to recover 
>>> and repair it (see the journal-related commands in Graylog) or simply 
>>> remove the journal files from disk.
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Monday, 4 April 2016 22:56:47 UTC+2, kluch wrote:

 After "out of disk space" I removed journals, old indices, cycled 
 deflector and then even deleted all indices from ES but it not helped at 
 all. All graylog nodes restarted and still nothing.
 I dont know from where graylog reads/displays negative (with minus) 
 value of unprocessed messages.  On web it looks like this:
 Processing *1* incoming and *0* outgoing msg/s. *-1,195,856,763 
 unprocessed messages* are currently in the journal, in 1 segments. *1 
 messages* have been appended to, and *0 messages* have been read from 
 the journal in the last second.
 Moreover graylog does not write to elasticsearch so only unprocessed 
 messages counter is changing. Time is in sync with ntp. Version 1.3.4 
 from OVA.
 Any help?

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2718c30f-08d5-42e6-8137-f82ab7b01460%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: VMware OVF disk filled, now have all shards unassigned

[Obie]

1,841,685 unprocessed messages are currently in the journal, in 13 segments.
11 messages have been appended in the last second, 0 messages have been 
read in the last second.


On Friday, August 26, 2016 at 9:45:39 AM UTC-4, Obie wrote:
>
> Ok, so I read that you wouldn't want to assign them when you only have one 
> node.
>
> Messages are coming in, but no searches return anything. What needs to be 
> done here? Some sort of re-index?
>
> Thanks
>
>
>
> On Friday, August 26, 2016 at 9:34:53 AM UTC-4, Obie wrote:
>>
>> How can I get these assigned even though I only have one node?
>>
>> Thanks
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/6c4aa0f2-e1e6-4c39-8457-a970df442ffe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog not writing to elasticsearch after out of disk space, ES green but...

[Obie]

Some commands or a link to a doc would be helpful here.

On Thursday, August 4, 2016 at 12:53:46 PM UTC-4, juli...@gmail.com wrote:
>
> Ok so what are those commands?? Can you provide a link or something? 
> Googling what you mention isn't very explicit and results aren't helpful. 
> Thx
>
> On Tuesday, 5 April 2016 04:02:08 UTC-4, Jochen Schalanda wrote:
>>
>> Hi,
>>
>> it looks like your journal is corrupted. You can either try to recover 
>> and repair it (see the journal-related commands in Graylog) or simply 
>> remove the journal files from disk.
>>
>> Cheers,
>> Jochen
>>
>> On Monday, 4 April 2016 22:56:47 UTC+2, kluch wrote:
>>>
>>> After "out of disk space" I removed journals, old indices, cycled 
>>> deflector and then even deleted all indices from ES but it not helped at 
>>> all. All graylog nodes restarted and still nothing.
>>> I dont know from where graylog reads/displays negative (with minus) 
>>> value of unprocessed messages.  On web it looks like this:
>>> Processing *1* incoming and *0* outgoing msg/s. *-1,195,856,763 
>>> unprocessed messages* are currently in the journal, in 1 segments. *1 
>>> messages* have been appended to, and *0 messages* have been read from 
>>> the journal in the last second.
>>> Moreover graylog does not write to elasticsearch so only unprocessed 
>>> messages counter is changing. Time is in sync with ntp. Version 1.3.4 
>>> from OVA.
>>> Any help?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/41f736a3-28d7-440f-9d4d-ca61f8e1416d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: VMware OVF disk filled, now have all shards unassigned

[Obie]

Ok, so I read that you wouldn't want to assign them when you only have one 
node.

Messages are coming in, but no searches return anything. What needs to be 
done here? Some sort of re-index?

Thanks



On Friday, August 26, 2016 at 9:34:53 AM UTC-4, Obie wrote:
>
> How can I get these assigned even though I only have one node?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/da3dd3f9-91e7-4543-87f9-d7e779053862%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: VMware OVF disk filled, now have all shards unassigned

[Obie]

How can I get these assigned even though I only have one node?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/0ae42b83-785e-401d-9918-6ca488ec14a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: graylog2 trimmed mean percentage?

[Yiannis Karayiannidis]

Hi Jan,
I was talking about a function on field values.
It would great

Regards


2016-08-22 18:24 GMT+03:00 Jan Doberstein :

> Hej Yiannis,
>
> Any help with that ?
>
> Sorry what you like to get is currently not possible. Fixed values would
> be possible to trim with pipelines. But you would like to have this
> dynamic. That is not possible. Maybe in the future, but that sounds like a
> function on field values. I do not know any open source product that have
> this available.
>
>
> with kind regards
>
> Jan
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/graylog2/0gZHOtxi2V8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/etPan.57bb193e.175c10b.7f2c%40jalogisch.de
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMbCpVpeSSVUQYF7sKYsiVuB5TU_SX%3DN8c%3D2Px3zKFyWT0MMrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: VMware OVF disk filled, now have all shards unassigned

[Obie]

Thanks... I ruled out the scripting part, issued the following and got a 
response:

ubuntu@graylog:~$ curl -XPOST 'host:9200/_cluster/reroute' -d '{
> "commands" : [ {
>"allocate" : {
>   "index" : "graylog_0",
>   "shard" : "0",
>   "node" : "Atum",
>   "allow_primary" : true
> }
>   }
> ]
>   }'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[allocate]
 
allocation of [graylog_0][0] on node 
{Atum}{U6oodK5vRpeql3-Ld5py6g}{ipaddr}{ipaddr:9300} is not allowed, reason: 
[YES(node passes include/exclude/require filters)][YES(shard not primary or 
relocation disabled)][YES(primary is already active)][YES(below shard 
recovery limit of [2])][YES(only a single data node is present)][YES(total 
shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(allocation 
disabling is ignored)][YES(target node version [2.3.1] is same or newer 
than source node version [2.3.1])][YES(no allocation awareness 
enabled)][NO(shard cannot be allocated on same node 
[U6oodK5vRpeql3-Ld5py6g] it already exists on)][YES(allocation disabling is 
ignored)]"}],"type":"illegal_argument_exception","reason":"[allocate] 
allocation of [graylog_0][0] on node 
{Atum}{U6oodK5vRpeql3-Ld5py6g}{ipaddr}{ipaddr:9300} is not allowed, reason: 
[YES(node passes include/exclude/require filters)][YES(shard not primary or 
relocation disabled)][YES(primary is already active)][YES(below shard 
recovery limit of [2])][YES(only a single data node is present)][YES(total 
shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(allocation 
disabling is ignored)][YES(target node version [2.3.1] is same or newer 
than source node version [2.3.1])][YES(no allocation awareness 
enabled)][NO(shard cannot be allocated on same node 
[U6oodK5vRpeql3-Ld5py6g] it already exists on)][YES(allocation disabling is 
ignored)]"},"status":400}ubuntu@graylog:~$

Thanks






On Thursday, August 25, 2016 at 12:45:57 PM UTC-4, Obie wrote:
>
> Hello,
>
> I'm running the VMware OVF and the root partition filled. I increased it 
> and all services start properly, but now all Elasticsearch shards are 
> unassigned.
>
>   Elasticsearch cluster is yellow. Shards: 12 active, 0 initializing, 0
>  relocating, 12 unassigned, What does this mean? 
> 
>
> I read a post about deleting the alias deflector target but am not sure 
> how to do this, nor am I sure if this is the recommended fix.
>
> Any guidance?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4a213d1b-2199-4a27-948a-c8f0870c9b90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Multiple nodes in a cluster

[Steve Kuntz]

Thank You!

Something so simple... embarrassed I didn't find it myself.

On Friday, August 26, 2016 at 6:50:54 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Steve,
>
> On Wednesday, 24 August 2016 22:55:21 UTC+2, Steve Kuntz wrote:
>>
>> [NodePingThread] Did not find meta info of this node. Re-registering. I 
>> have changed all IPs appropriately in the configuration of the 3rd node.
>>
>
> This error message is usually a sign of clock skew on the Graylog nodes 
> (use NTP or a similar technology to keep clocks in sync) or long GC pauses 
> in the Graylog processes (which should trigger a system notification you 
> can see on the System page in the Graylog web interface).
>
>
> Cheers,
> Jochen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/dbfb1a39-cf1f-4a90-99da-ad3b4254d69e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: JSON extractor in 2.0.3, am I missing something?

[Kostya Vasilyev]

Jochen,

On Friday, August 26, 2016 at 1:49:04 PM UTC+3, Jochen Schalanda wrote:
>
> Hi Kostya,
>
> On Thursday, 25 August 2016 16:02:23 UTC+2, Kostya Vasilyev wrote:
>>
>> If you meant that there are no double quotes around key names -- that's 
>> just how shows in the Graylog UI.
>>
>
> No, that's the actual content of the "result" field.
>


This: {subs=57, devs=34} is not valid JSON, I agree.

But this: {"subs": 57, "devs": 34} is valid JSON, this was in the data 
returns by my HTTP API.

Oh, right, it's not just the quotes, it's the "=" vs. ":" key / value 
delimiters.

Sorry I somehow missed it.

But this "string representation of a dictionary" was produced by Graylog's 
"HTTP JSON Input", presumably there is a reason why this format was picked.

And so the issue really is, to take a step back -- how can I work with this 
data (inside "result")?
 

>
>
> On Thursday, 25 August 2016 16:02:23 UTC+2, Kostya Vasilyev wrote:
>>
>> Or maybe I'm wrong about "that's just how shows in the Graylog UI", and 
>> Graylog's "http JSON" input loses double quotes around nested keys? That 
>> would be a bug, wouldn't it?
>>
>
> The content of the "result" field isn't JSON at all but a string 
> representation of the extracted map/dictionary. If you consider this a bug, 
> please create an issue for this at 
> https://github.com/Graylog2/graylog2-server/issues.
>
>
Before I do that, let me try again, from a broader perspective.

I've got an HTTP API that returns some values as a JSON object.

Then I've configured a "HTTP JSON input" in Graylog, pointing to this HTTP 
API.

As we now know (thanks again), this records the "result" as a string that's 
not JSON, rather it's a "Java toString" or similar, and can't be processed 
with the JSON extractor.

Now the question:

How can extract data from "result" into individual values?

One solution would be to have the input itself do that, using a more 
"specific" JSON selector (path) of the actual (single!) value to extract.

But then I'd need to create essentially same exact inputs for this API, one 
for each value that I need. Not very convenient.

Two possible solutions that I can think of would be:

1 - Being able to specify several, not one, JSON selectors in an HTTP JSON 
input, and the name of a message property for each one.

Something like this:

_subs = $.objs.subs
_devs = $.objs.devs
_db_write_ops = $.db.stats.write.op_count

I guess that's not possible right now?

2 - Maybe there is an extractor -- not the JSON extractor but some other 
kind -- which already is able to work with what I've got in "result" right 
now?

That would be able to extract the values of "devs" and "subs" and store 
them as message properties?

I can't seem to find one like that, am I missing something?

-- K

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/14980621-9765-4b4f-9944-1fc2bd09c124%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Multiple nodes in a cluster

[Jochen Schalanda]

Hi Steve,

On Wednesday, 24 August 2016 22:55:21 UTC+2, Steve Kuntz wrote:
>
> [NodePingThread] Did not find meta info of this node. Re-registering. I 
> have changed all IPs appropriately in the configuration of the 3rd node.
>

This error message is usually a sign of clock skew on the Graylog nodes 
(use NTP or a similar technology to keep clocks in sync) or long GC pauses 
in the Graylog processes (which should trigger a system notification you 
can see on the System page in the Graylog web interface).


Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/bffc5cb7-725c-47a1-bb17-0ec1a2844ca0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: JSON extractor in 2.0.3, am I missing something?

[Jochen Schalanda]

Hi Kostya,

On Thursday, 25 August 2016 16:02:23 UTC+2, Kostya Vasilyev wrote:
>
> If you meant that there are no double quotes around key names -- that's 
> just how shows in the Graylog UI.
>

No, that's the actual content of the "result" field.


On Thursday, 25 August 2016 16:02:23 UTC+2, Kostya Vasilyev wrote:
>
> Or maybe I'm wrong about "that's just how shows in the Graylog UI", and 
> Graylog's "http JSON" input loses double quotes around nested keys? That 
> would be a bug, wouldn't it?
>

The content of the "result" field isn't JSON at all but a string 
representation of the extracted map/dictionary. If you consider this a bug, 
please create an issue for this at 
https://github.com/Graylog2/graylog2-server/issues.


Cheers,
Jochen

>
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/73abe625-cc4f-448d-b1fd-ef8290470e52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Autmatically parsed fields in Syslog TCP/UDP input

[Jan Doberstein]

Hej Markus,


I filed an enhancement in github ( 
https://github.com/Graylog2/graylog2-server/issues/2739 ) but that was closed 
quickly with the "tip" to just use Raw Text Input - which isn't a solution 
because that Input is lacking Syslog fields i need ( level, facility, ... ).
take a look what other already do: 
https://marketplace.graylog.org/addons?search=vmware 

personal i would use the „raw input“ and extract the fields i like to see with 
the give groks in the marketplace - or take a google / github search. I was 
able to identify different resources that can help you with this.



/jd

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/etPan.57c01b36.bb31434.ae4f%40jalogisch.de.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Message signed with OpenPGP using AMPGpg

[graylog2] Autmatically parsed fields in Syslog TCP/UDP input

[Markus Fischbacher]

Heja,

the Syslog inputs - i just use TCP/UDP but i think all working the same - 
extracting at least two fields ( application_name and process_id ) 
automatically. The problem hear is, that not all message are that well 
formed - ESXi, SAN in my case. I haven't found a way to disable that out of 
the box in graylog. My current workaround is to extract the fields with an 
extractor manually but even there i can't handle all variants. A pipeline 
to remove the fields doesn't work either because that would remove those 
manual extracted fields too.

I filed an enhancement in github ( 
https://github.com/Graylog2/graylog2-server/issues/2739 ) but that was 
closed quickly with the "tip" to just use Raw Text Input - which isn't a 
solution because that Input is lacking Syslog fields i need ( level, 
facility, ... ).

I thought about making a plugin with a modified Syslog UDP input but before 
that i hope to get some other views. From my point of view, the default 
Graylog Syslog Inputs should have a option to disable those additional 
fields.

Any ideas instead? 

Best regards,
MArkus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2bfe529c-0c7d-49dc-82e7-837ff80595e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: grok exporter Problem... or a bug

[Markus Fischbacher]

Arrgh...

finally solved it - just needed a sleep. Solution was to split the OR for 
the first part of the string then the field and the last part of the string 
as another or.

ESXI_PID (((\: cpu\d+:)|(\[))%{POSINT:process_id}((\))|(\]:)))|(\:)

Don't think it's a bug ;-) The online validators solved the double assigned 
field but only in an array - so even there it wasn't clean. My fault.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d6ff5200-038e-46de-84e7-409fc5d1dcca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: How to change the data type of an extracted field

[Gray Log]

If anyone should stumble here, I was able to force solve it by manually 
cycling the deflector and keeping the query range within the creation time 
of the new index. This may (probably) cause issues later but in proof of 
concept phase, it doesn't matter. I have finally been able to win over 
management - the PoC is a success.

On Friday, August 26, 2016 at 5:22:50 PM UTC+10, Jochen Schalanda wrote:
>
> Hi,
>
> you can't change the types of a field in an existing index. The schema has 
> to be defined up-front.
>

I couldn't find that statement anywhere in the documentation. Or even words 
to it's effect.

But I believe it would aide newcomers significantly if it _was_more clearly 
stated, especially w.r.t. getting the VM up and running and producing what 
management want to see from a limited time pilot now - rather than the 
"Wait till tomorrow boss. And hope we don't make the same mistake again, 
else you'll have to wait till the next day...lather, rinse, repeat" 
scenario.  That makes it hard to sell and makes mgmt believe other products 
are more likely to be capable of (what they perceive to be) such a simple 
thing.


> I'm pretty sure you want to read 
> http://docs.graylog.org/en/2.0/pages/configuration/elasticsearch.html#custom-index-mappings
> .
>

Reading that without knowing the former point is, hmm, not overly helpful 
when trying to get a POC up and running quickly. Obviously there is a lot 
of domain knowledge to gain which is tricky at the POC stage, like you MUST 
click that Add button next to the conversion in the extractor creation. 
Thanks for trying to help anyway.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/909aa885-bd14-4f96-9c97-e0ad48e399cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: How to change the data type of an extracted field

[Jochen Schalanda]

Hi,

you can't change the types of a field in an existing index. The schema has 
to be defined up-front.

I'm pretty sure you want to read 
http://docs.graylog.org/en/2.0/pages/configuration/elasticsearch.html#custom-index-mappings
.

Cheers,
Jochen

On Friday, 26 August 2016 05:38:44 UTC+2, Gray Log wrote:
>
> Hi,
>
> (Using current 2.0.3 OVA)
>
> I created an extractor on the default syslog-udp input and despite setting 
> the conversion to Numeric, I neglected to click the Add button.  Thus the 
> field was created as type string.
>
> Thus it cannot be graphed because it is a string. Quick values are not 
> sufficient, I need the graph (and statistical calcs also).
>
> Following other guides I have removed all instances of the offending 
> field, then I deleted the extractor completely, then I re-created it.
>
> However, the new values are *still* strings even though I deleted those 
> fields and the extractor previously.
>
>   "tcp_seq_num" : {
> "type" : "string",
> "index" : "not_analyzed"
>   },
>
> So, how do you change the field type of an already created field, right 
> now? ie. not after the indexes are rotated or at any other time, but 
> immediately, right now.
>
>
> https://github.com/Graylog2/graylog2-web-interface/issues/1592#issuecomment-137448785
> "One solution for the problem is to wait: once your ES indices are 
> rotated, the removed fields will go away. If that's not good enough for 
> your case, you can manually delete them in Elasticsearch."
>
>
>
> Given that I have deleted them from elasticsearch, then why do they remain 
> as strings afterwards?  What is the correct process?
>
> For reference, this is what I did:
> 1) Created the extractor without clicking the Add button next to the 
> conversion drop down.
> 2) Logs are received and the new field is created and appears on left hand 
> menu.
>
> All looks great at this point until you try to graph the result - only 
> then do you discover your mistake. And now all those collected logs are 
> useless it would appear.
>
> 3) Delete the extractor to stop it creating more bogus data
> 4) Hunt down and delete every field: (see here for details: 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html
> )
>
> $ curl 
> 'localhost:9200/graylog_0/_search?q=_exists_:tcp_seq_num=id' 
> | grep _id | cut -d\" -f4  | while read id;do \
> echo "curl -XPOST \"localhost:9200/graylog_0/message/${id}/_update\" -d '{ 
> \"script\" : \"ctx._source.remove(\\\"pf_tcp_seq_num\\\")\" }'"; \
> done > delme
> $ sh delme
>
> Note Well: I do it in 2 stages just for convenience, create the script 
> "delme" containing the commands then execute them "sh delme".
>
> 5) Run the search again to make sure they were deleted: curl 
> 'localhost:9200/graylog_0/_search?q=_exists_:tcp_seq_num=id' 
> => No results
> 6) Recreate the extractor with correct numeric conversion applied.
> 7) Wait for logs to arrive.
> 8) Try to graph the field -> error, cannot graph strings. WTF?
> 9) Re-examine mappings - curl -X GET '
> http://localhost:9200/graylog_0/_mappings?pretty'
>
> It is identical to the way it was before:
>
>   "tcp_seq_num" : {
> "type" : "string",
> "index" : "not_analyzed"
>   },
>
> Surely, there must be a way that I'm missing.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/543de41d-4a7b-454e-94c7-6200c5bfd262%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: [ANNOUNCE] Graylog v2.1.0-RC.1 has been released

[Jochen Schalanda]

Hi,

On Thursday, 25 August 2016 23:33:54 UTC+2, 123Dev wrote:
>
> Or is it simply following these steps.
>
> http://docs.graylog.org/en/latest/pages/configuration/graylog_ctl.html#upgrade-graylog
> Which were used to upgrade from 2.0 to 2.0.3
>

Yes, that should still work but make sure to read through 
https://github.com/Graylog2/graylog2-server/blob/2.1.0-rc.1/UPGRADING.rst 
before upgrading.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/195a325b-afe1-4ee2-89c8-bf281bdc826d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: VMware OVF disk filled, now have all shards unassigned

[Jochen Schalanda]

Hi Obie,

On Thursday, 25 August 2016 23:03:42 UTC+2, Obie wrote:
>
> I tried this script:
>
>
> #!/bin/bash
>
> for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep 
> UNASSIGNED | awk '{print $2}'); do
>   curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
> "commands" : [ {
>"allocate" : {
>   "index" : "graylog_0",
>   "shard" : $shard,
>   "node" : "Atum",
>   "allow_primary" : true
> }
>   }
> ]
>   }'
>   sleep 5
>
> done
>
>
> And get this error:
>
>
> ubuntu@graylog:~$ ./shards_allocate.sh
>   % Total% Received % Xferd  Average Speed   TimeTime Time 
>  Current
>  Dload  Upload   Total   SpentLeft 
>  Speed
> 100  1440  100  14400 0260  0  0:00:05  0:00:05 --:--:--   
> 358
> {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unrecognized 
> token '$shard': was expecting ('true', 'false' or 'null')\n at [Source: 
> org.elasticsearch.transport.netty.ChannelBufferStreamInput@8227be3; line: 
> 5, column: 28]"}],"type":"json_parse_exception","reason":"Unrecognized 
> token '$shard': was expecting ('true', 'false' or 'null')\n at [Source: 
> org.elasticsearch.transport.netty.ChannelBufferStreamInput@8227be3; line: 
> 5, column: 28]"},"status":500}^C
> ubuntu@graylog:~$
>
>
> What am I missing?
>

Single quotes don't allow string interpolation. You have to use double 
quotes and escape the double quotes inside the string 
accordingly: http://wiki.bash-hackers.org/syntax/quoting

Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/09c6e194-7792-4685-89c2-a516a411598f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.