Re: [LEDE-DEV] [PATCH ubox v2] Add template support to logread

[Henry Chang]

Hi Petr,

> From my point of view and experience, logd is good for simple use cases. For
anything else I would recommend you to use syslogd from Busybox for example.

The reason I chose to stay with logd is I wanna keep basing on ubus.

> syslogd from Busybox handles log rotation very well, including multiple old
files. AFAIK logd doesn't support more then one rotated file.

I don't need file rotation. If I need, I can simply use logrotate.

> What about log uploading over HTTPS, using curl and a few lines of shell
script?  I bet, that some of those cloud log services support such input
source, you can tag and format it as needed also.

This is definitely doable. And this is also what I'm doing currently.
I use stunnel for securing the connection. The thing is, string
parsing is more expensive than object parsing. So if we can do object
parsing, why not?

> It seems to me, that you're just trying to bend actual and simple tools for
your use case, instead of using right tools for the job.

I didn't change anything in logd, instead, I added a customizable
output for its reader logread. There's a regex filtering feature in
logread, I think the reason it's there is very similar to why I added
this template.

I can write a separate reader for logd to match my own use case. But
while I was thinking of that, I also came up an idea that I should
make it more general so that future people don't have to worry about
the same stuff. And I really don't think this is a special case.

Regards,

Henry

On Tue, May 2, 2017 at 3:11 AM, Petr Štetiar  wrote:
> Henry Chang  [2017-04-28 14:14:57]:
>
>> 1. logd is built-in logd is a built-in feature of ubox, and ubox is a heart
>> of OpenWrt/LEDE. If ubox can achieve all the use cases, we should simply
>> just use it. Use rsyslog instead will create additional footprint in terms
>> of resource while a lot of features in rsyslog is not going to be used.
>
> From my point of view and experience, logd is good for simple use cases. For
> anything else I would recommend you to use syslogd from Busybox for example.
>
>> "Simple plain text files also require log rotation to prevent them
>> from becoming too large. In log rotation, existing log files are
>
> syslogd from Busybox handles log rotation very well, including multiple old
> files. AFAIK logd doesn't support more then one rotated file.
>
>> 3. To make a working and secure remote logging pipeline, what we need
>> is just logread, stunnel, and a log server. A log server will probably
>> need some sort of special format in order to do the authentication or
>> classification. So without template feature in logread, we'll need
>> another service in between logread and stunnel for just manipulating
>> every line of the log or we write another logread to achieve this.
>> While logread already has some sort of object of the log, doing this
>> in logread will be easier for both programmers and the computer.
>
> What about log uploading over HTTPS, using curl and a few lines of shell
> script?  I bet, that some of those cloud log services support such input
> source, you can tag and format it as needed also.
>
>> 4. The template feature in this patch is very efficient. It doesn't
>> create any overhead if the user doesn't enable this feature by passing
>> the corresponding argument. The string replacement is also efficient,
>> we can improve it by optimizing some strlen() function calls for sure
>> but basically it's efficient enough.
>
> It seems to me, that you're just trying to bend actual and simple tools for
> your use case, instead of using right tools for the job.
>
> -- ynezz

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] [PATCH ubox v2] Add template support to logread

[Petr Štetiar]

Henry Chang  [2017-04-28 14:14:57]:

> 1. logd is built-in logd is a built-in feature of ubox, and ubox is a heart
> of OpenWrt/LEDE. If ubox can achieve all the use cases, we should simply
> just use it. Use rsyslog instead will create additional footprint in terms
> of resource while a lot of features in rsyslog is not going to be used.

>From my point of view and experience, logd is good for simple use cases. For
anything else I would recommend you to use syslogd from Busybox for example.

> "Simple plain text files also require log rotation to prevent them
> from becoming too large. In log rotation, existing log files are

syslogd from Busybox handles log rotation very well, including multiple old
files. AFAIK logd doesn't support more then one rotated file.

> 3. To make a working and secure remote logging pipeline, what we need
> is just logread, stunnel, and a log server. A log server will probably
> need some sort of special format in order to do the authentication or
> classification. So without template feature in logread, we'll need
> another service in between logread and stunnel for just manipulating
> every line of the log or we write another logread to achieve this.
> While logread already has some sort of object of the log, doing this
> in logread will be easier for both programmers and the computer.

What about log uploading over HTTPS, using curl and a few lines of shell
script?  I bet, that some of those cloud log services support such input
source, you can tag and format it as needed also.

> 4. The template feature in this patch is very efficient. It doesn't
> create any overhead if the user doesn't enable this feature by passing
> the corresponding argument. The string replacement is also efficient,
> we can improve it by optimizing some strlen() function calls for sure
> but basically it's efficient enough.

It seems to me, that you're just trying to bend actual and simple tools for
your use case, instead of using right tools for the job.

-- ynezz

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] [PATCH ubox v2] Add template support to logread

[Henry Chang]

Hi,

Since I got some questions from the subscribers, I'd like to explain
more why I wanted to add this feature to a cutdown logging system logd
instead of using a more comprehensive syslog implementation such as
rsyslog.

1. logd is built-in
logd is a built-in feature of ubox, and ubox is a heart of
OpenWrt/LEDE. If ubox can achieve all the use cases, we should simply
just use it. Use rsyslog instead will create additional footprint in
terms of resource while a lot of features in rsyslog is not going to
be used.

2. rsyslog lacks ring buffer
logd has a very good ring buffer architecture, and it also comes with
a very good client logread. Deal with the log files created by rsyslog
is kind of complex, as you can see some examples here:
https://www.loggly.com/blog/why-journald/

"Simple plain text files also require log rotation to prevent them
from becoming too large. In log rotation, existing log files are
renamed and compressed. Any programs that watch syslog messages for
problems have to deal with this somehow."

3. To make a working and secure remote logging pipeline, what we need
is just logread, stunnel, and a log server. A log server will probably
need some sort of special format in order to do the authentication or
classification. So without template feature in logread, we'll need
another service in between logread and stunnel for just manipulating
every line of the log or we write another logread to achieve this.
While logread already has some sort of object of the log, doing this
in logread will be easier for both programmers and the computer.

4. The template feature in this patch is very efficient. It doesn't
create any overhead if the user doesn't enable this feature by passing
the corresponding argument. The string replacement is also efficient,
we can improve it by optimizing some strlen() function calls for sure
but basically it's efficient enough.

Regards,

Henry

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] [PATCH ubox v2] Add template support to logread

[Henry Chang]

According to the documentation, it needs a token as a credential in
every single line of log.

For example, the template for rsyslog is "<%pri%>%protocol-version%
%timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid%
[TOKEN@41058 tag=\"TAG\"] %msg%\n"

https://www.loggly.com/docs/rsyslog-manual-configuration/

On Thu, Apr 27, 2017 at 10:20 PM, Baptiste Jonglez
 wrote:
> Hi,
>
> On Thu, Apr 27, 2017 at 04:33:31PM -0700, Henry Chang wrote:
>> Hi,
>>
>> I would like to integrate logd with a cloud logging service.
>> The service only accepts certain format of log, so I decided to make logread 
>> support an output template.
>
> Can't this service use standard syslog messages?
>
> E.g. by using something like:
>
>   config system
>   option log_remote '1'
>   option log_ip '42.42.42.42'
>
>> Here's the usage:
>>
>> logread -T "%priority% %source% %message% %timestamp%"
>>
>> Currently supports 4 keywords: priority, source, message, timestamp.
>> The keywords should be surrounded by percent signs as showed above.
>>
>> Regards,
>>
>> Henry Chang
>>
>> Henry Chang (1):
>>   logread: Add support for output template
>>
>>  log/logread.c | 153 
>> --
>>  1 file changed, 127 insertions(+), 26 deletions(-)

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] [PATCH ubox v2] Add template support to logread

[Baptiste Jonglez]

Hi,

On Thu, Apr 27, 2017 at 04:33:31PM -0700, Henry Chang wrote:
> Hi,
> 
> I would like to integrate logd with a cloud logging service.
> The service only accepts certain format of log, so I decided to make logread 
> support an output template.

Can't this service use standard syslog messages?

E.g. by using something like:

  config system
  option log_remote '1'
  option log_ip '42.42.42.42'

> Here's the usage:
> 
> logread -T "%priority% %source% %message% %timestamp%"
> 
> Currently supports 4 keywords: priority, source, message, timestamp.
> The keywords should be surrounded by percent signs as showed above.
> 
> Regards,
> 
> Henry Chang
> 
> Henry Chang (1):
>   logread: Add support for output template
> 
>  log/logread.c | 153 
> --
>  1 file changed, 127 insertions(+), 26 deletions(-)


signature.asc
Description: PGP signature
___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

[LEDE-DEV] [PATCH ubox v2] Add template support to logread

[Henry Chang]

From: Henry Chang 

Hi,

I would like to integrate logd with a cloud logging service.
The service only accepts certain format of log, so I decided to make logread 
support an output template.

Here's the usage:

logread -T "%priority% %source% %message% %timestamp%"

Currently supports 4 keywords: priority, source, message, timestamp.
The keywords should be surrounded by percent signs as showed above.

Regards,

Henry Chang

Henry Chang (1):
  logread: Add support for output template

 log/logread.c | 153 --
 1 file changed, 127 insertions(+), 26 deletions(-)

-- 
2.7.4


___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev