Re: grep: show matching line from pattern file
Hello, of course, there are different ways to solve this, i like the perl approach. Only since i myself am not all that familiar with the language, i'd like to add 2 pointers: (M)AWK scripting language can do similar things (read syslog once, loop over regular expressions and output anything you want about it). But if you can live with calling egrep repeatedly, i would suggest GNU parallel, which works similar to xargs, only a much enhanced version of it, using sevral cores in parallel by default but also handling the commandline in a much improved way (special syntax, so to speak). It allows coding your request as a one-liner, i am certain, but probably not as effective, as perl or awk would have been. BTW: GNU parallel is in debian repos, but a quite outdated version of it. Have fun, DdB
Re: grep: show matching line from pattern file
On 3/06/22 07:17, Greg Wooledge wrote: On Thu, Jun 02, 2022 at 03:12:23PM -0400, duh wrote: > > Jim Popovitch wrote on 28/05/2022 21:40: > > > I have a file of regex patterns and I use grep like so: > > > > > > ~$ grep -f patterns.txt /var/log/syslog > > > > > > What I'd like to get is a listing of all lines, specifically the line > > > numbers of the regexps in patterns.txt, that match entries in > > > /var/log/syslog. Is there a way to do this? $cat -n /var/log/syslog | grep warn and it found "warn" in the syslog file and provided line numbers. I have not used the -f option You're getting the line numbers from the log file. The OP wanted the line numbers of the patterns in the -f pattern file. Why? I have no idea. There is no standard option to do this, because it's not a common requirement. That's why I wrote one from scratch in perl. I don't know what the OP's use case is, but here's an example I might use: I have a bunch of custom ignore files for logcheck. After a software upgrade, I might want to check which patterns no longer match anything, and can be deleted or modified. I'd really still want to check with real egrep, though, rather than using perl's re engine instead. Cheers, Richard
Re: grep: show matching line from pattern file
* David Christensen [22-06/02=Thu 15:50 -0700]:
> On 6/2/22 15:13, Will Mengarini wrote:
>> * Greg Wooledge [22-05/28=Sa 17:11 -0400]:
>>> [...]
>>> #!/usr/bin/perl
>>> use strict; use warnings;
>>> [...]
>>> open PATS, ">> [...]
>>
>> You need "or die", not "|| die", because of precedence: what you coded
>> checks whether "> wanted to check whether the result of open() is logically true.
>
> +1 That is a good explanation of a Perl fine point/ gotcha.
>
>> In this transcript, the number before the prompt-ending '$' is $?:
>>
>> debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
>> debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
>> Died at -e line 1.
>> debian/pts/4 bash3 ~ 14:57 2$
>>
>
> What is your shell? PS1?
The shell is Bash 5.1.4. My PS1 is constructed by an elaborate script
that's old enough to have sex in Thailand, but you can get the effect
of what I posted by setting PS1 with the line
PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
assuming you're running at least Bash 2.05a. You may prefer
PS1="\\h/${TTY#/dev/} \\s^$SHLVL \\w \\A \$?\\\$"
.
My original script was coded for Bash 1.4.7, and had to do
PS1="\\h${TTY#/dev/} \\s$SHLVL \\w \`s=\$?;date +%H:%M;exit \$s\` \$?\\\$"
because \A wasn't available, so 'date' had to be run in a subshell
that needed to take care to save and restore $?. (The variable
it uses for that, s, goes away when the subshell does; and that
scary-looking exit just exits the subshell, resetting $?.)
--
Will Mengarini
Free software: the Source will be with you, always.
sh -c 'echo -n MENGARINI|sum -s|colrm 4'
Re: grep: show matching line from pattern file
On 6/2/22 15:13, Will Mengarini wrote: * Greg Wooledge [22-05/28=Sa 17:11 -0400]: [...] #!/usr/bin/perl use strict; use warnings; [...] open PATS, " You need "or die", not "|| die", because of precedence: what you coded checks whether " +1 That is a good explanation of a Perl fine point/ gotcha. In this transcript, the number before the prompt-ending '$' is $?: debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die' debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die' Died at -e line 1. debian/pts/4 bash3 ~ 14:57 2$ What is your shell? PS1? David
Re: grep: show matching line from pattern file
* Greg Wooledge [22-05/28=Sa 17:11 -0400]: > [...] > #!/usr/bin/perl > use strict; use warnings; > [...] > open PATS, " [...] You need "or die", not "|| die", because of precedence: what you coded checks whether " perl -le"print unpack '%C*',MENGARINI"
Re: grep: show matching line from pattern file
On Thu, Jun 02, 2022 at 03:12:23PM -0400, duh wrote: > > > Jim Popovitch wrote on 28/05/2022 21:40: > > > > I have a file of regex patterns and I use grep like so: > > > > > > > > ~$ grep -f patterns.txt /var/log/syslog > > > > > > > > What I'd like to get is a listing of all lines, specifically the line > > > > numbers of the regexps in patterns.txt, that match entries in > > > > /var/log/syslog. Is there a way to do this? > $cat -n /var/log/syslog | grep warn > > and it found "warn" in the syslog file and provided line numbers. I have > not used the -f option You're getting the line numbers from the log file. The OP wanted the line numbers of the patterns in the -f pattern file. Why? I have no idea. There is no standard option to do this, because it's not a common requirement. That's why I wrote one from scratch in perl.
Re: grep: show matching line from pattern file
On 5/29/22 9:44 AM, David Wright wrote: On Sun 29 May 2022 at 15:02:35 (+0200), Jörg-Volker Peetz wrote: Jim Popovitch wrote on 28/05/2022 21:40: Not exactly Debian specific, but hoping that someone here can help. I have a file of regex patterns and I use grep like so: ~$ grep -f patterns.txt /var/log/syslog What I'd like to get is a listing of all lines, specifically the line numbers of the regexps in patterns.txt, that match entries in /var/log/syslog. Is there a way to do this? How about this: $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt That will only work for literal patterns, not regex ones. Cheers, David. I may be missing a lot but I will risk throwing in my 2 cents (which is no longer worth the proverbial amount with the current inflation) $cat -n /var/log/syslog | grep warn and it found "warn" in the syslog file and provided line numbers. I have not used the -f option (but am now aware of it in the pastfrom your post -- thank you. Might file that away for future use but will not press my luck at the moment). My approach saves me the effort of writing a scriptl. Ahh, the Perl philosophy of "there is more than one way to do it".
Re: grep: show matching line from pattern file
On Sat, 2022-05-28 at 17:11 -0400, Greg Wooledge wrote:
> On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
> > On 2022-05-28 at 15:40, Jim Popovitch wrote:
> > > I have a file of regex patterns and I use grep like so:
> > >
> > > ~$ grep -f patterns.txt /var/log/syslog
> > >
> > > What I'd like to get is a listing of all lines, specifically the line
> > > numbers of the regexps in patterns.txt, that match entries in
> > > /var/log/syslog. Is there a way to do this?
> >
> > I don't know of a standardized way to do that (if anyone else wants to
> > suggest one, I'm open to learn), but of course it *can* be done, via
> > scripting. Off the top of my head, I came up with the following
> >
> > for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
> > if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
> > /dev/null ; then
> > echo $line ;
> > fi
> > done
>
> The quoting here is... completely absent (and that's extremely bad), but
> also importantly, one would ideally like to avoid running grep a thousand
> times, especially if the target logfile is large.
>
> I believe this is the kind of job for which perl is well-suited. I'm not
> great at perl, but I'll give it a shot.
>
> Here's a version with some extra information as output, so I can verify
> that it's doing something reasonably close to correct:
>
>
> #!/usr/bin/perl
> use strict; use warnings;
>
> my @patlist;
> open PATS, " chomp(@patlist = );
> close PATS;
>
> while (<>) {
> chomp;
> for (my $i = 0; $i <= $#patlist; $i++) {
> print "$i|$patlist[$i]|$_\n" if /$patlist[$i]/;
> }
> }
>
>
> Now, to test it, we need a patterns.txt file:
>
>
> unicorn:~$ cat patterns.txt
> PATH
> HOME|~
> a...e
>
>
> And an input (log) file:
>
>
> unicorn:~$ cat file
> zebra
> Home, home on the range.
> Oops, I meant HOME on the range.
>
> applesauce
>
>
> And here's what it does:
>
>
> unicorn:~$ ./foo file
> 1|HOME|~|Oops, I meant HOME on the range.
> 2|a...e|applesauce
>
>
> Pattern numbers 1 and 2 (the second and third, since it starts at 0) were
> matched, so we have a line for each of those.
>
> If that's kinda what you wanted, then you can adjust this to do precisely
> what you wanted. It shouldn't take a lot of work, I hope. Well, I guess
> that depends on what you really want.
>
> Bash is not well-suited to this task, and even if we were to take The
> Wanderer's script and fix all the issues in it, it would still be a
> vastly inferior solution. Some tools are just not meant for some jobs.
>
Thanks Greg, that is exactly what I needed, and double thanks for the
details in explaining it, etc.
-Jim P.
Re: grep: show matching line from pattern file
On Sun 29 May 2022 at 15:02:35 (+0200), Jörg-Volker Peetz wrote: > Jim Popovitch wrote on 28/05/2022 21:40: > > Not exactly Debian specific, but hoping that someone here can help. > > > > I have a file of regex patterns and I use grep like so: > > > > ~$ grep -f patterns.txt /var/log/syslog > > > > What I'd like to get is a listing of all lines, specifically the line > > numbers of the regexps in patterns.txt, that match entries in > > /var/log/syslog. Is there a way to do this? > > How about this: > > $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt That will only work for literal patterns, not regex ones. Cheers, David.
Re: grep: show matching line from pattern file
Jim Popovitch wrote on 28/05/2022 21:40: Not exactly Debian specific, but hoping that someone here can help. I have a file of regex patterns and I use grep like so: ~$ grep -f patterns.txt /var/log/syslog What I'd like to get is a listing of all lines, specifically the line numbers of the regexps in patterns.txt, that match entries in /var/log/syslog. Is there a way to do this? -Jim P. How about this: $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt Regards, Jörg.
Re: grep: show matching line from pattern file
On 2022-05-28 at 17:11, Greg Wooledge wrote: > On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote: > >> On 2022-05-28 at 15:40, Jim Popovitch wrote: >> > I have a file of regex patterns and I use grep like so: >> > >> >~$ grep -f patterns.txt /var/log/syslog >> > >> > What I'd like to get is a listing of all lines, specifically the line >> > numbers of the regexps in patterns.txt, that match entries in >> > /var/log/syslog. Is there a way to do this? >> >> I don't know of a standardized way to do that (if anyone else wants to >> suggest one, I'm open to learn), but of course it *can* be done, via >> scripting. Off the top of my head, I came up with the following >> >> for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do >> if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog > >> /dev/null ; then >> echo $line ; >> fi >> done > > The quoting here is... completely absent (and that's extremely bad), but > also importantly, one would ideally like to avoid running grep a thousand > times, especially if the target logfile is large. A brother of mine has schooled me on several things I did wrong here already, and I am now going over my long-tail stockpile of scripts with shellcheck, seeing what I can learn. (I normally do quote variables, but for some reason it slipped my mind this time until after I'd already hit Send. Some of these are known-safe values which won't need quoting, but others aren't, so the principle remains valid to cite.) This wasn't especially intended as a great solution, in any case; it started out as me trying to write shell-like pseudocode, but then I couldn't see any obvious reason why it wouldn't work, and when I tried it out I only needed a few tweaks before it did. There's a *reason* I had the "YMMV" on that message. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
Re: grep: show matching line from pattern file
On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
> On 2022-05-28 at 15:40, Jim Popovitch wrote:
> > I have a file of regex patterns and I use grep like so:
> >
> >~$ grep -f patterns.txt /var/log/syslog
> >
> > What I'd like to get is a listing of all lines, specifically the line
> > numbers of the regexps in patterns.txt, that match entries in
> > /var/log/syslog. Is there a way to do this?
>
> I don't know of a standardized way to do that (if anyone else wants to
> suggest one, I'm open to learn), but of course it *can* be done, via
> scripting. Off the top of my head, I came up with the following
>
> for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
> if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
> /dev/null ; then
> echo $line ;
> fi
> done
The quoting here is... completely absent (and that's extremely bad), but
also importantly, one would ideally like to avoid running grep a thousand
times, especially if the target logfile is large.
I believe this is the kind of job for which perl is well-suited. I'm not
great at perl, but I'll give it a shot.
Here's a version with some extra information as output, so I can verify
that it's doing something reasonably close to correct:
#!/usr/bin/perl
use strict; use warnings;
my @patlist;
open PATS, ");
close PATS;
while (<>) {
chomp;
for (my $i = 0; $i <= $#patlist; $i++) {
print "$i|$patlist[$i]|$_\n" if /$patlist[$i]/;
}
}
Now, to test it, we need a patterns.txt file:
unicorn:~$ cat patterns.txt
PATH
HOME|~
a...e
And an input (log) file:
unicorn:~$ cat file
zebra
Home, home on the range.
Oops, I meant HOME on the range.
applesauce
And here's what it does:
unicorn:~$ ./foo file
1|HOME|~|Oops, I meant HOME on the range.
2|a...e|applesauce
Pattern numbers 1 and 2 (the second and third, since it starts at 0) were
matched, so we have a line for each of those.
If that's kinda what you wanted, then you can adjust this to do precisely
what you wanted. It shouldn't take a lot of work, I hope. Well, I guess
that depends on what you really want.
Bash is not well-suited to this task, and even if we were to take The
Wanderer's script and fix all the issues in it, it would still be a
vastly inferior solution. Some tools are just not meant for some jobs.
Re: grep: show matching line from pattern file
On 2022-05-28 at 15:40, Jim Popovitch wrote: > Not exactly Debian specific, but hoping that someone here can help. > > I have a file of regex patterns and I use grep like so: > >~$ grep -f patterns.txt /var/log/syslog > > What I'd like to get is a listing of all lines, specifically the line > numbers of the regexps in patterns.txt, that match entries in > /var/log/syslog. Is there a way to do this? I don't know of a standardized way to do that (if anyone else wants to suggest one, I'm open to learn), but of course it *can* be done, via scripting. Off the top of my head, I came up with the following for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog > /dev/null ; then echo $line ; fi done I just tested that on my own system, with a different file (since I'm not root right now) and a couple of exact-string patterns found by examining that file, and it seems to work as intended. YMMV. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
