subject:"\[rsyslog\] about imfile"

Re: [rsyslog] about imfile

2016-12-01 Thread Rainer Gerhards

2016-12-01 13:09 GMT+01:00 mosto...@gmail.com :
> now, that makes sense! :D
>

Sorry for the initial confusion. That was so obvious to me that I even
didn't think it was worth mentioning. Of course it's not obvious ;-)
But that's also the reason why I say I am not the best person to write
the user doc. I appreciate your help here very much!

Rainer
> Thanks
>
>
> El 01/12/16 a las 13:06, Rainer Gerhards escribió:
>>
>> 2016-12-01 12:55 GMT+01:00 Rainer Gerhards :
>>>
>>> 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com :
>
> because a syslog message contains tag.

 mind-blowing explanation :P
>>>
>>> Well, as the property is already there, why would you like to have a
>>> config parameter for something that by definition will never be
>>> needed?
>>
>> A, I think I just understand where we have the misunderstanding:
>>
>> im(p)tcp by definition processes syslog messages
>> imfile by definition processes text file lines (which are NOT syslog
>> messages)
>> so im(p)tcp always has a tag, and hence needs no config parama
>> where imfile by definition does not have a tag and thus needs one
>> configured.
>>
>> Does that help?
>> Rainer
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com




A message without TAG (malformed RFC 3164 message), no matter if it's read
from file or it arrives from socket, won't have a tag
Hence, setting it only for imfile won't fix it for socket modules.

I am not ready for this discussion again. In rsyslog, rfc3164 messages
always have a tag. See previous lengthy discussions ;-)

You misread my message, but your other response just solved the thread ;)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com


now, that makes sense! :D

Thanks


El 01/12/16 a las 13:06, Rainer Gerhards escribió:

2016-12-01 12:55 GMT+01:00 Rainer Gerhards :

2016-12-01 11:54 GMT+01:00 mosto...@gmail.com :

because a syslog message contains tag.

mind-blowing explanation :P

Well, as the property is already there, why would you like to have a
config parameter for something that by definition will never be
needed?

A, I think I just understand where we have the misunderstanding:

im(p)tcp by definition processes syslog messages
imfile by definition processes text file lines (which are NOT syslog messages)
so im(p)tcp always has a tag, and hence needs no config parama
where imfile by definition does not have a tag and thus needs one configured.

Does that help?
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread Rainer Gerhards

2016-12-01 13:06 GMT+01:00 mosto...@gmail.com :
> El 01/12/16 a las 12:55, Rainer Gerhards escribió:
>>
>> 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com :

 because a syslog message contains tag.
>>>
>>> mind-blowing explanation :P
>>
>> Well, as the property is already there, why would you like to have a
>> config parameter for something that by definition will never be
>> needed?
>
> A RFC 3164 formatted message contains a tag, no matter if it's read from
> file or it arrives from socket.
> Hence, there's no need to have a TAG property for any of them.

mails crossed

>
> A message without TAG (malformed RFC 3164 message), no matter if it's read
> from file or it arrives from socket, won't have a tag
> Hence, setting it only for imfile won't fix it for socket modules.

I am not ready for this discussion again. In rsyslog, rfc3164 messages
always have a tag. See previous lengthy discussions ;-)

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread Rainer Gerhards

2016-12-01 12:55 GMT+01:00 Rainer Gerhards :
> 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com :
>>> because a syslog message contains tag.
>>
>> mind-blowing explanation :P
>
> Well, as the property is already there, why would you like to have a
> config parameter for something that by definition will never be
> needed?

A, I think I just understand where we have the misunderstanding:

im(p)tcp by definition processes syslog messages
imfile by definition processes text file lines (which are NOT syslog messages)
so im(p)tcp always has a tag, and hence needs no config parama
where imfile by definition does not have a tag and thus needs one configured.

Does that help?
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com


El 01/12/16 a las 12:55, Rainer Gerhards escribió:

2016-12-01 11:54 GMT+01:00 mosto...@gmail.com :

because a syslog message contains tag.

mind-blowing explanation :P

Well, as the property is already there, why would you like to have a
config parameter for something that by definition will never be
needed?
A RFC 3164 formatted message contains a tag, no matter if it's read from 
file or it arrives from socket.

Hence, there's no need to have a TAG property for any of them.

A message without TAG (malformed RFC 3164 message), no matter if it's 
read from file or it arrives from socket, won't have a tag

Hence, setting it only for imfile won't fix it for socket modules.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread Rainer Gerhards

2016-12-01 11:54 GMT+01:00 mosto...@gmail.com :
>> because a syslog message contains tag.
>
> mind-blowing explanation :P

Well, as the property is already there, why would you like to have a
config parameter for something that by definition will never be
needed?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com




read modes other than 0 currently seem to have issues in inotify mode

   Any open issues? it's an based-on-experienced-warning message? legacy?



I am not aware of one, which does not necessarily mean none exists. So
you need to check the issue trackers :-(

The longer-term question is if we should grandfather readMode. The
performance difference seems not to be much, and a single approach is
much better to maintain.
I won't remove that until it has been confirmed or not if that's 
actually an issue.








no, because other input modules don't hard-code these values, they set
them based on the message they receive. It doesn't make sense to have them
apply to all modules.

I don't understand your reasoning here.
Why it makes sense to set tag when using imfile but not with imtcp?

because a syslog message contains tag.

mind-blowing explanation :P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread Rainer Gerhards

2016-12-01 11:20 GMT+01:00 mosto...@gmail.com :
> El 30/11/16 a las 22:51, David Lang escribió:
>>
>> On Wed, 30 Nov 2016, mosto...@gmail.com wrote:
>>> read modes other than 0 currently seem to have issues in inotify mode
>>>
>>>   Any open issues? it's an based-on-experienced-warning message? legacy?
>>
>>
>> good question
>
> Rainer?

I am not aware of one, which does not necessarily mean none exists. So
you need to check the issue trackers :-(

The longer-term question is if we should grandfather readMode. The
performance difference seems not to be much, and a single approach is
much better to maintain.

>
>>
>>> imfile has tag, facility and severity properties...
>>>
>>>   Is there any way this properties being /inherited/ for ALL modules?
>>>   (hence documented on "/input-modules/")
>>
>>
>> no, because other input modules don't hard-code these values, they set
>> them based on the message they receive. It doesn't make sense to have them
>> apply to all modules.
>
> I don't understand your reasoning here.
> Why it makes sense to set tag when using imfile but not with imtcp?

because a syslog message contains tag.

>
>>
>>> @radu-gheorghe @rgerhards could you have a look at
>>> https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst

will try to look asap, but pretty busy witjh code and design at the moment...

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread David Lang


On Thu, 1 Dec 2016, mosto...@gmail.com wrote:

Note that when $WorkDirectory is not set or set to a non-writable 
location, the state file **will not be generated**.


  Am I wrong or state files are written to / in this scenario?


no, without a work directory set, they don't get written to /. As the doc 
says, they just don't get written anywhere.
This is not what is happening on my tests. Setting WorkDirectory to 
non-existing directory make it create imfile-state on /. Just opened an 
issue.


slightly different than not setting it at all :-) good catch on this bug.


read modes other than 0 currently seem to have issues in inotify mode

  Any open issues? it's an based-on-experienced-warning message? legacy?


good question

Rainer?




imfile has tag, facility and severity properties...

  Is there any way this properties being /inherited/ for ALL modules?
  (hence documented on "/input-modules/")


no, because other input modules don't hard-code these values, they set them 
based on the message they receive. It doesn't make sense to have them apply 
to all modules.

I don't understand your reasoning here.
Why it makes sense to set tag when using imfile but not with imtcp?


all messages arriving via imtcp are supposed to be in rfc-3164 or rfc-5424, both 
of which set all these values.


imfile is the only input to let you set them, because it's the only input where 
you don't get the data from the log source. When you are reading data from 
arbitrary log files from a program, they don't have these properties in the 
file.


One drawback with imfile is that if the file has this data in it, it's hard to 
make use of it.



trimlineoverbytes should actually apply to all modes, why only to some?

I'm just a monkey typing...ask someone who knows!


is reopen on truncate really still experimental?

It was marked so...


Rainer?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com

El 30/11/16 a las 22:51, David Lang escribió:

On Wed, 30 Nov 2016, mosto...@gmail.com wrote:

According to documentation:

State files are used to track which parts of the monitored file are
already processed.

Do state files keep just "last reading position" or as doc suggests
a file can be processed in multiple chunks(parts)?

I'd have to look at the format of the state file to be absolutly sure,
but I think it just keeps track of the last reasing position. I don't
think you can have multiple threads reading the same file, so if you
read a file in chunks, each time you read a chunk it advances the
position.

I asked 'cause documentation wasn't clear enough for me.

Note that when $WorkDirectory is not set or set to a non-writable
location, the state file **will not be generated**.

Am I wrong or state files are written to / in this scenario?

no, without a work directory set, they don't get written to /. As the
doc says, they just don't get written anywhere.
This is not what is happening on my tests. Setting WorkDirectory to
non-existing directory make it create imfile-state on /. Just opened an
issue.

Regarding pollinginterval: During each polling interval, all files
are processed in a round-robin fashion.

I'm confused. Does this mean files are readed, sleep for X seconds,
and readed again...
or rsyslog reads documents during X seconds looping in a round-robin
fashion?

the first.

Thanks

readtimeout: This can be used with *startmsg.regex* (but not *readMode*)

Why it can't be used with readmode? (Apart from obviously not
implemented)

just not implemented (I actually expected that it would be implemented
for readmodes)

read modes other than 0 currently seem to have issues in inotify mode

Any open issues? it's an based-on-experienced-warning message? legacy?

good question

Rainer?

imfile has tag, facility and severity properties...

Is there any way this properties being /inherited/ for ALL modules?
(hence documented on "/input-modules/")

no, because other input modules don't hard-code these values, they set
them based on the message they receive. It doesn't make sense to have
them apply to all modules.

I don't understand your reasoning here.
Why it makes sense to set tag when using imfile but not with imtcp?

@radu-gheorghe @rgerhards could you have a look at
https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst

my comments

re: examples needed TODOs, are these items really needed? It seems to
me that the explinations are pretty clear, I could see examples adding
as much confusion as clarification.

Ok.

re: windows/inode, this documentation is about the unix version. the
windows version is slightly different (it has a GUI amoung other
things), and it isn't free.

it's not always clear why you have TODO there. In most cases, the text
following the TODO seems appropriate, could you change this to either
put the description of what needs to change on it's own line, or
otherwise indicate what needs to be changed?

TODO=>WIP :P

I would group all the EXPERT options in one section, with the big
warning at the top of them that if you don't understand them you
should not set them.

LGTM

I would also add a warning that they almost never need to be changed,
even on high load systems, so benchmarks should be run before and
after changing any of them because they sometimes have non-intuitive
performance impact.

I would not set escapelf as an expert option, but rather make a
grouping of options under the category "dealing with multi-line logs"
and put it there along with readmode, regex.startmsg and the related
timeouts.

Ok.

trimlineoverbytes should actually apply to all modes, why only to some?

I'm just a monkey typing...ask someone who knows!

is reopen on truncate really still experimental?

It was marked so...

I would put the depriciated items in their own section.

Done

Thank you a lot David.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

Re: [rsyslog] about imfile

2016-11-30 Thread David Lang

On Wed, 30 Nov 2016, mosto...@gmail.com wrote:

According to documentation:

State files are used to track which parts of the monitored file are already
processed.

Do state files keep just "last reading position" or as doc suggests
a file can be processed in multiple chunks(parts)?

I'd have to look at the format of the state file to be absolutly sure, but I
think it just keeps track of the last reasing position. I don't think you can
have multiple threads reading the same file, so if you read a file in chunks,
each time you read a chunk it advances the position.

Note that when $WorkDirectory is not set or set to a non-writable location,
the state file **will not be generated**.

Am I wrong or state files are written to / in this scenario?

no, without a work directory set, they don't get written to /. As the doc says,
they just don't get written anywhere.

Regarding pollinginterval: During each polling interval, all files are
processed in a round-robin fashion.

I'm confused. Does this mean files are readed, sleep for X seconds,
and readed again...
or rsyslog reads documents during X seconds looping in a round-robin
fashion?

the first.

readtimeout: This can be used with *startmsg.regex* (but not *readMode*)

Why it can't be used with readmode? (Apart from obviously not
implemented)

just not implemented (I actually expected that it would be implemented for
readmodes)

read modes other than 0 currently seem to have issues in inotify mode

Any open issues? it's an based-on-experienced-warning message? legacy?

good question

imfile has tag, facility and severity properties...

Is there any way this properties being /inherited/ for ALL modules?
(hence documented on "/input-modules/")

no, because other input modules don't hard-code these values, they set them
based on the message they receive. It doesn't make sense to have them apply to
all modules.

@radu-gheorghe @rgerhards could you have a look at
https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst

my comments

re: examples needed TODOs, are these items really needed? It seems to me that
the explinations are pretty clear, I could see examples adding as much confusion
as clarification.

re: windows/inode, this documentation is about the unix version. the windows
version is slightly different (it has a GUI amoung other things), and it isn't
free.

it's not always clear whay you have TODO there. In most cases, the text
following the TODO seems appropriate, could you change this to either put the
description of what needs to change on it's own line, or otherwise indicate what
needs to be changed?

I would group all the EXPERT options in one section, with the big warning at the
top of them that if you don't understand them you should not set them.

I would also add a warning that they almost never need to be changed, even on
high load systems, so benchmarks should be run before and after changing any of
them because they sometimes have non-intuitive performance impact.

I would not set escapelf as an expert option, but rather make a grouping of
options under the category "dealing with multi-line logs" and put it there along
with readmode, regex.startmsg and the related timeouts.

trimlineoverbytes should actually apply to all modes, why only to some?

is reopen on truncate really still experimental?

I would put the depriciated items in their own section.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

[rsyslog] about imfile

2016-11-30 Thread mosto...@gmail.com