Re: [rsyslog] how to remove the blank space in $!all-json string?

David Lang Mon, 25 May 2015 11:56:01 -0700

On Mon, 25 May 2015, chenlin rao wrote:

yeah, not only $!all-json.
For example. I have a logline:


```
{"content":"Traceroute Result of 'api.weibo.cn':\nDNS is
'202.103.224.68'\nIP is
'180.149.153.216'\n1|192:168:1:1|34.482ms\n2|219:159:136:1|27.005ms\n3|218:65:201:21|206.549ms\n4|202:103:236:53|116.733ms\n5|*\n6|*\n7|*\n8|180:149:128:54|339.948ms\n9|*\n10|180:149:129:178|302.180ms\n11|180:149:153:216|1.840s\n\n","__date":1432483892.9061}
```

My template is

```
template( name="local6JsonArray" type="list" ) {
   constant(value="{\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
   constant(value="\",\"host\":\"") property(name="hostname")
   constant(value="\",") property(name="$.line" position.from="2")
}
```

The _source JSON in ES was:

```
{"@timestamp":"2015-05-25T07:29:35+08:00","host":"
web032.mweibo.yf.sinanode.com", "content": "Traceroute Result of '
api.weibo.cn':\nDNS is '202.103.224.68'\nIP is
'180.149.153.216'\n1|192:168:1:1|34.482ms\n2|219:159:136:1|27.005ms\n3|218:65:201:21|206.549ms\n4|202:103:236:53|116.733ms\n5|*\n6|*\n7|*\n8|180:149:128:54|339.948ms\n9|*\n10|180:149:129:178|302.180ms\n11|180:149:153:216|1.840s\n\n",
"__date": 1432477827.545300 }
```

There are blank spaces around ", :, }.

ElasticSearch does not store the raw JSON, it first parses the JSON and thendeals with the various parts. As such, whitespace in the JSON should not takeany more space. And the amount of extra network bandwidth needed should beunmeasurable in practice.


David Lang


2015-05-25 17:35 GMT+08:00 Rainer Gerhards <[email protected]>:

sample please. But I need to say that $!all-json is pretty fixed...

2015-05-25 11:31 GMT+02:00 chenlin rao <[email protected]>:

Hi all.
    Can we remove the blank space inside the json strings?
    I like to use mmjsonparse and omelasticsearch plugins. But the json
strings contain too many blank space, so we need more net flow, and
elasticsearch would store the blank space bit in it's _source JSON which
means more disk size.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] how to remove the blank space in $!all-json string?

Reply via email to