Re: [rsyslog] about imfile
2016-12-01 13:09 GMT+01:00 mosto...@gmail.com : > now, that makes sense! :D > Sorry for the initial confusion. That was so obvious to me that I even didn't think it was worth mentioning. Of course it's not obvious ;-) But that's also the reason why I say I am not the best person to write the user doc. I appreciate your help here very much! Rainer > Thanks > > > El 01/12/16 a las 13:06, Rainer Gerhards escribió: >> >> 2016-12-01 12:55 GMT+01:00 Rainer Gerhards : >>> >>> 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : > > because a syslog message contains tag. mind-blowing explanation :P >>> >>> Well, as the property is already there, why would you like to have a >>> config parameter for something that by definition will never be >>> needed? >> >> A, I think I just understand where we have the misunderstanding: >> >> im(p)tcp by definition processes syslog messages >> imfile by definition processes text file lines (which are NOT syslog >> messages) >> so im(p)tcp always has a tag, and hence needs no config parama >> where imfile by definition does not have a tag and thus needs one >> configured. >> >> Does that help? >> Rainer >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
A message without TAG (malformed RFC 3164 message), no matter if it's read from file or it arrives from socket, won't have a tag Hence, setting it only for imfile won't fix it for socket modules. I am not ready for this discussion again. In rsyslog, rfc3164 messages always have a tag. See previous lengthy discussions ;-) You misread my message, but your other response just solved the thread ;) ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
now, that makes sense! :D Thanks El 01/12/16 a las 13:06, Rainer Gerhards escribió: 2016-12-01 12:55 GMT+01:00 Rainer Gerhards : 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : because a syslog message contains tag. mind-blowing explanation :P Well, as the property is already there, why would you like to have a config parameter for something that by definition will never be needed? A, I think I just understand where we have the misunderstanding: im(p)tcp by definition processes syslog messages imfile by definition processes text file lines (which are NOT syslog messages) so im(p)tcp always has a tag, and hence needs no config parama where imfile by definition does not have a tag and thus needs one configured. Does that help? Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
2016-12-01 13:06 GMT+01:00 mosto...@gmail.com : > El 01/12/16 a las 12:55, Rainer Gerhards escribió: >> >> 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : because a syslog message contains tag. >>> >>> mind-blowing explanation :P >> >> Well, as the property is already there, why would you like to have a >> config parameter for something that by definition will never be >> needed? > > A RFC 3164 formatted message contains a tag, no matter if it's read from > file or it arrives from socket. > Hence, there's no need to have a TAG property for any of them. mails crossed > > A message without TAG (malformed RFC 3164 message), no matter if it's read > from file or it arrives from socket, won't have a tag > Hence, setting it only for imfile won't fix it for socket modules. I am not ready for this discussion again. In rsyslog, rfc3164 messages always have a tag. See previous lengthy discussions ;-) Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
2016-12-01 12:55 GMT+01:00 Rainer Gerhards : > 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : >>> because a syslog message contains tag. >> >> mind-blowing explanation :P > > Well, as the property is already there, why would you like to have a > config parameter for something that by definition will never be > needed? A, I think I just understand where we have the misunderstanding: im(p)tcp by definition processes syslog messages imfile by definition processes text file lines (which are NOT syslog messages) so im(p)tcp always has a tag, and hence needs no config parama where imfile by definition does not have a tag and thus needs one configured. Does that help? Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
El 01/12/16 a las 12:55, Rainer Gerhards escribió: 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : because a syslog message contains tag. mind-blowing explanation :P Well, as the property is already there, why would you like to have a config parameter for something that by definition will never be needed? A RFC 3164 formatted message contains a tag, no matter if it's read from file or it arrives from socket. Hence, there's no need to have a TAG property for any of them. A message without TAG (malformed RFC 3164 message), no matter if it's read from file or it arrives from socket, won't have a tag Hence, setting it only for imfile won't fix it for socket modules. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : >> because a syslog message contains tag. > > mind-blowing explanation :P Well, as the property is already there, why would you like to have a config parameter for something that by definition will never be needed? Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
read modes other than 0 currently seem to have issues in inotify mode Any open issues? it's an based-on-experienced-warning message? legacy? I am not aware of one, which does not necessarily mean none exists. So you need to check the issue trackers :-( The longer-term question is if we should grandfather readMode. The performance difference seems not to be much, and a single approach is much better to maintain. I won't remove that until it has been confirmed or not if that's actually an issue. no, because other input modules don't hard-code these values, they set them based on the message they receive. It doesn't make sense to have them apply to all modules. I don't understand your reasoning here. Why it makes sense to set tag when using imfile but not with imtcp? because a syslog message contains tag. mind-blowing explanation :P ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
2016-12-01 11:20 GMT+01:00 mosto...@gmail.com : > El 30/11/16 a las 22:51, David Lang escribió: >> >> On Wed, 30 Nov 2016, mosto...@gmail.com wrote: >>> read modes other than 0 currently seem to have issues in inotify mode >>> >>> Any open issues? it's an based-on-experienced-warning message? legacy? >> >> >> good question > > Rainer? I am not aware of one, which does not necessarily mean none exists. So you need to check the issue trackers :-( The longer-term question is if we should grandfather readMode. The performance difference seems not to be much, and a single approach is much better to maintain. > >> >>> imfile has tag, facility and severity properties... >>> >>> Is there any way this properties being /inherited/ for ALL modules? >>> (hence documented on "/input-modules/") >> >> >> no, because other input modules don't hard-code these values, they set >> them based on the message they receive. It doesn't make sense to have them >> apply to all modules. > > I don't understand your reasoning here. > Why it makes sense to set tag when using imfile but not with imtcp? because a syslog message contains tag. > >> >>> @radu-gheorghe @rgerhards could you have a look at >>> https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst will try to look asap, but pretty busy witjh code and design at the moment... Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
On Thu, 1 Dec 2016, mosto...@gmail.com wrote: Note that when $WorkDirectory is not set or set to a non-writable location, the state file **will not be generated**. Am I wrong or state files are written to / in this scenario? no, without a work directory set, they don't get written to /. As the doc says, they just don't get written anywhere. This is not what is happening on my tests. Setting WorkDirectory to non-existing directory make it create imfile-state on /. Just opened an issue. slightly different than not setting it at all :-) good catch on this bug. read modes other than 0 currently seem to have issues in inotify mode Any open issues? it's an based-on-experienced-warning message? legacy? good question Rainer? imfile has tag, facility and severity properties... Is there any way this properties being /inherited/ for ALL modules? (hence documented on "/input-modules/") no, because other input modules don't hard-code these values, they set them based on the message they receive. It doesn't make sense to have them apply to all modules. I don't understand your reasoning here. Why it makes sense to set tag when using imfile but not with imtcp? all messages arriving via imtcp are supposed to be in rfc-3164 or rfc-5424, both of which set all these values. imfile is the only input to let you set them, because it's the only input where you don't get the data from the log source. When you are reading data from arbitrary log files from a program, they don't have these properties in the file. One drawback with imfile is that if the file has this data in it, it's hard to make use of it. trimlineoverbytes should actually apply to all modes, why only to some? I'm just a monkey typing...ask someone who knows! is reopen on truncate really still experimental? It was marked so... Rainer? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
El 30/11/16 a las 22:51, David Lang escribió: On Wed, 30 Nov 2016, mosto...@gmail.com wrote: According to documentation: State files are used to track which parts of the monitored file are already processed. Do state files keep just "last reading position" or as doc suggests a file can be processed in multiple chunks(parts)? I'd have to look at the format of the state file to be absolutly sure, but I think it just keeps track of the last reasing position. I don't think you can have multiple threads reading the same file, so if you read a file in chunks, each time you read a chunk it advances the position. I asked 'cause documentation wasn't clear enough for me. Note that when $WorkDirectory is not set or set to a non-writable location, the state file **will not be generated**. Am I wrong or state files are written to / in this scenario? no, without a work directory set, they don't get written to /. As the doc says, they just don't get written anywhere. This is not what is happening on my tests. Setting WorkDirectory to non-existing directory make it create imfile-state on /. Just opened an issue. Regarding pollinginterval: During each polling interval, all files are processed in a round-robin fashion. I'm confused. Does this mean files are readed, sleep for X seconds, and readed again... or rsyslog reads documents during X seconds looping in a round-robin fashion? the first. Thanks readtimeout: This can be used with *startmsg.regex* (but not *readMode*) Why it can't be used with readmode? (Apart from obviously not implemented) just not implemented (I actually expected that it would be implemented for readmodes) Ok read modes other than 0 currently seem to have issues in inotify mode Any open issues? it's an based-on-experienced-warning message? legacy? good question Rainer? imfile has tag, facility and severity properties... Is there any way this properties being /inherited/ for ALL modules? (hence documented on "/input-modules/") no, because other input modules don't hard-code these values, they set them based on the message they receive. It doesn't make sense to have them apply to all modules. I don't understand your reasoning here. Why it makes sense to set tag when using imfile but not with imtcp? @radu-gheorghe @rgerhards could you have a look at https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst my comments re: examples needed TODOs, are these items really needed? It seems to me that the explinations are pretty clear, I could see examples adding as much confusion as clarification. Ok. re: windows/inode, this documentation is about the unix version. the windows version is slightly different (it has a GUI amoung other things), and it isn't free. Ok it's not always clear why you have TODO there. In most cases, the text following the TODO seems appropriate, could you change this to either put the description of what needs to change on it's own line, or otherwise indicate what needs to be changed? TODO=>WIP :P I would group all the EXPERT options in one section, with the big warning at the top of them that if you don't understand them you should not set them. LGTM I would also add a warning that they almost never need to be changed, even on high load systems, so benchmarks should be run before and after changing any of them because they sometimes have non-intuitive performance impact. I would not set escapelf as an expert option, but rather make a grouping of options under the category "dealing with multi-line logs" and put it there along with readmode, regex.startmsg and the related timeouts. Ok. trimlineoverbytes should actually apply to all modes, why only to some? I'm just a monkey typing...ask someone who knows! is reopen on truncate really still experimental? It was marked so... I would put the depriciated items in their own section. Done Thank you a lot David. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] about imfile
On Wed, 30 Nov 2016, mosto...@gmail.com wrote: According to documentation: State files are used to track which parts of the monitored file are already processed. Do state files keep just "last reading position" or as doc suggests a file can be processed in multiple chunks(parts)? I'd have to look at the format of the state file to be absolutly sure, but I think it just keeps track of the last reasing position. I don't think you can have multiple threads reading the same file, so if you read a file in chunks, each time you read a chunk it advances the position. Note that when $WorkDirectory is not set or set to a non-writable location, the state file **will not be generated**. Am I wrong or state files are written to / in this scenario? no, without a work directory set, they don't get written to /. As the doc says, they just don't get written anywhere. Regarding pollinginterval: During each polling interval, all files are processed in a round-robin fashion. I'm confused. Does this mean files are readed, sleep for X seconds, and readed again... or rsyslog reads documents during X seconds looping in a round-robin fashion? the first. readtimeout: This can be used with *startmsg.regex* (but not *readMode*) Why it can't be used with readmode? (Apart from obviously not implemented) just not implemented (I actually expected that it would be implemented for readmodes) read modes other than 0 currently seem to have issues in inotify mode Any open issues? it's an based-on-experienced-warning message? legacy? good question imfile has tag, facility and severity properties... Is there any way this properties being /inherited/ for ALL modules? (hence documented on "/input-modules/") no, because other input modules don't hard-code these values, they set them based on the message they receive. It doesn't make sense to have them apply to all modules. @radu-gheorghe @rgerhards could you have a look at https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst my comments re: examples needed TODOs, are these items really needed? It seems to me that the explinations are pretty clear, I could see examples adding as much confusion as clarification. re: windows/inode, this documentation is about the unix version. the windows version is slightly different (it has a GUI amoung other things), and it isn't free. it's not always clear whay you have TODO there. In most cases, the text following the TODO seems appropriate, could you change this to either put the description of what needs to change on it's own line, or otherwise indicate what needs to be changed? I would group all the EXPERT options in one section, with the big warning at the top of them that if you don't understand them you should not set them. I would also add a warning that they almost never need to be changed, even on high load systems, so benchmarks should be run before and after changing any of them because they sometimes have non-intuitive performance impact. I would not set escapelf as an expert option, but rather make a grouping of options under the category "dealing with multi-line logs" and put it there along with readmode, regex.startmsg and the related timeouts. trimlineoverbytes should actually apply to all modes, why only to some? is reopen on truncate really still experimental? I would put the depriciated items in their own section. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.