Re: grep: show matching line from pattern file
Hello, of course, there are different ways to solve this, i like the perl approach. Only since i myself am not all that familiar with the language, i'd like to add 2 pointers: (M)AWK scripting language can do similar things (read syslog once, loop over regular expressions and output anything you want about it). But if you can live with calling egrep repeatedly, i would suggest GNU parallel, which works similar to xargs, only a much enhanced version of it, using sevral cores in parallel by default but also handling the commandline in a much improved way (special syntax, so to speak). It allows coding your request as a one-liner, i am certain, but probably not as effective, as perl or awk would have been. BTW: GNU parallel is in debian repos, but a quite outdated version of it. Have fun, DdB
Re: grep: show matching line from pattern file
On 3/06/22 07:17, Greg Wooledge wrote: On Thu, Jun 02, 2022 at 03:12:23PM -0400, duh wrote: > > Jim Popovitch wrote on 28/05/2022 21:40: > > > I have a file of regex patterns and I use grep like so: > > > > > > ~$ grep -f patterns.txt /var/log/syslog > > > > > > What I'd like to get is a listing of all lines, specifically the line > > > numbers of the regexps in patterns.txt, that match entries in > > > /var/log/syslog. Is there a way to do this? $cat -n /var/log/syslog | grep warn and it found "warn" in the syslog file and provided line numbers. I have not used the -f option You're getting the line numbers from the log file. The OP wanted the line numbers of the patterns in the -f pattern file. Why? I have no idea. There is no standard option to do this, because it's not a common requirement. That's why I wrote one from scratch in perl. I don't know what the OP's use case is, but here's an example I might use: I have a bunch of custom ignore files for logcheck. After a software upgrade, I might want to check which patterns no longer match anything, and can be deleted or modified. I'd really still want to check with real egrep, though, rather than using perl's re engine instead. Cheers, Richard
Re: Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
On 6/2/22 22:50, Will Mengarini wrote:
* David Christensen [22-06/02=Th 19:18 -0700]:
[...]
Now I can almost match your prompt -- there is a dash before 'bash':
2022-06-02 19:05:10 dpchrist@laalaa ~
$ PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
laalaa/pts/8 -bash1 ~ 19:08 0$
The dash seems to be coming from the '\s' bash(1) -> PROMPTING ->
backslash-escaped special characters:
2022-06-02 19:12:58 dpchrist@laalaa ~
$ PS1="\\s"
-bash
The dash indicates you're running a login shell. That's useful
information, because a login shell is initialized differently.
See section INVOCATION in `man bash`.
You see in the rendered prompt it says "-bash1"; "1" is $SHLVL.
If you run another bash from inside that bash, and again set PS1
as you did above, you should see "bash2" instead of "-bash1".
RTFM bash(1), I see:
PROMPTING
\s the name of the shell, the basename of $0 (the portion
following the final slash)
INVOCATION
A login shell is one whose first character of argument zero is a -,
or one started with the --login option.
Thank you for the explanation of your PS1 and the subtleties of Bash,
variable expansion, remove matching prefix pattern, $0, and PS1 '\s'.
David
Re: Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
* David Christensen [22-06/02=Th 19:18 -0700]:
> [...]
> Now I can almost match your prompt -- there is a dash before 'bash':
>
> 2022-06-02 19:05:10 dpchrist@laalaa ~
> $ PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
> laalaa/pts/8 -bash1 ~ 19:08 0$
>
> The dash seems to be coming from the '\s' bash(1) -> PROMPTING ->
> backslash-escaped special characters:
>
> 2022-06-02 19:12:58 dpchrist@laalaa ~
> $ PS1="\\s"
> -bash
The dash indicates you're running a login shell. That's useful
information, because a login shell is initialized differently.
See section INVOCATION in `man bash`.
You see in the rendered prompt it says "-bash1"; "1" is $SHLVL.
If you run another bash from inside that bash, and again set PS1
as you did above, you should see "bash2" instead of "-bash1".
Re: Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
On 6/2/22 19:25, Greg Wooledge wrote:
On Thu, Jun 02, 2022 at 06:01:11PM -0700, David Christensen wrote:
This is my PS1. '\u' does not work on all of Debian, FreeBSD, Cygwin, and
macOS, so the expansion of ${USER} is inserted between two string literals
when .profile runs and sets PS1:
2022-06-02 17:39:09 dpchrist@laalaa ~
$ grep PS1 .profile
export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
Variable expansions *are* performed when PS1 is evaluated. So, you
could simply do:
PS1='stuff $USER more stuff'
That will delay the expansion of USER (which by the way is a BSD-ism)
until the prompt is drawn. The way you've written it, $USER is
expanded at the time PS1 is assigned. Which is not wrong, so long as
the value of USER cannot change in the middle of a shell session...
but it's not how most of the experienced people would do it, I think.
I'm rather curious how you managed to find a system + bash version
where \u doesn't work. That sounds like something you'd want to report
to the bug-bash mailing list... or, possibly, a user error. \u should
work on any Unix-like system.
I found the reason for not using '\u' in Bash PS1 -- the 'toor' user on
FreeBSD:
2022-06-02 21:30:08 toor@f3 ~
# freebsd-version ; uname -a ; bash --version
12.3-RELEASE-p5
FreeBSD f3.tracy.holgerdanske.com 12.3-RELEASE-p5 FreeBSD
12.3-RELEASE-p5 GENERIC amd64
GNU bash, version 5.1.16(0)-release (amd64-portbld-freebsd12.3)
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
2022-06-02 21:30:16 toor@f3 ~
# egrep '^(root|toor)' /etc/passwd
root:*:0:0:Charlie &:/root:/bin/csh
toor:*:0:0:Bourne-again Superuser:/root:/usr/local/bin/bash
2022-06-02 21:30:28 toor@f3 ~
# whoami
root
2022-06-02 21:30:30 toor@f3 ~
# echo $USER
toor
2022-06-02 21:30:34 toor@f3 ~
# PS1='\n\D{%Y-%m-%d %H:%M:%S} \u@\h \w\n\$ '
2022-06-02 21:30:41 root@f3 ~
#
Your suggestion produces the desired prompt:
2022-06-02 21:30:41 root@f3 ~
# PS1='\n\D{%Y-%m-%d %H:%M:%S} $USER@\h \w\n\$ '
2022-06-02 21:31:29 toor@f3 ~
#
But, I think I will keep the braces:
2022-06-02 21:55:38 toor@f3 ~
# vi .profile
export PS1='\n\D{%Y-%m-%d %H:%M:%S} ${USER}@\h \w\n\$ '
2022-06-02 21:56:04 toor@f3 ~
# . .profile
2022-06-02 21:56:06 toor@f3 ~
# echo $PS1
\n\D{%Y-%m-%d %H:%M:%S} ${USER}@\h \w\n\$
2022-06-02 21:56:10 toor@f3 ~
#
(It works on all of my other platforms.)
Thank you,
David
Re: Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
On Thu, Jun 02, 2022 at 06:01:11PM -0700, David Christensen wrote:
> This is my PS1. '\u' does not work on all of Debian, FreeBSD, Cygwin, and
> macOS, so the expansion of ${USER} is inserted between two string literals
> when .profile runs and sets PS1:
>
> 2022-06-02 17:39:09 dpchrist@laalaa ~
> $ grep PS1 .profile
> export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
Variable expansions *are* performed when PS1 is evaluated. So, you
could simply do:
PS1='stuff $USER more stuff'
That will delay the expansion of USER (which by the way is a BSD-ism)
until the prompt is drawn. The way you've written it, $USER is
expanded at the time PS1 is assigned. Which is not wrong, so long as
the value of USER cannot change in the middle of a shell session...
but it's not how most of the experienced people would do it, I think.
I'm rather curious how you managed to find a system + bash version
where \u doesn't work. That sounds like something you'd want to report
to the bug-bash mailing list... or, possibly, a user error. \u should
work on any Unix-like system.
Re: Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
On 6/2/22 18:35, Will Mengarini wrote:
* David Christensen [22-06/02=Th 18:01 -0700]:
On 6/2/22 17:12, Will Mengarini wrote:
* David Christensen [22-06/02=Th 15:50 -0700]:
On 6/2/22 15:13, Will Mengarini wrote:
In this transcript, the number before the prompt-ending '$' is $?:
debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
Died at -e line 1.
debian/pts/4 bash3 ~ 14:57 2$
What is your shell? PS1?
The shell is Bash 5.1.4.
PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
The snippet '${TTY#/dev/}' seems to produce ' -' on my computer.
How does your computer produce 'pts/4 ' and what does it mean?
'pts/4' is an abbreviation of '/dev/pts/4', pseudoterminal 4.
TTY is `tty`; it's been so long I'd forgotten that's not available in
all shells. You should have the 'tty' program; it's in coreutils.
Is there a reason why you are using
double quotes, rather than single quotes?
So I can interpolate stuff like ${TTY#/dev/}. In your case,
you'll need to set TTY=`tty` before setting PS1, so Bash
can use string substitution to remove '/dev/' from it.
Okay. That explains TTY and bash(1) -> Parameter Expansion -> When not
performing substring expansion -> Remove matching prefix pattern:
2022-06-02 19:04:45 dpchrist@laalaa ~
$ TTY=`tty`
2022-06-02 19:04:53 dpchrist@laalaa ~
$ echo "'$TTY'"
'/dev/pts/8'
2022-06-02 19:04:58 dpchrist@laalaa ~
$ echo "'${TTY#/dev/}'"
'pts/8'
Now I can almost match your prompt -- there is a dash before 'bash':
2022-06-02 19:05:10 dpchrist@laalaa ~
$ PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
laalaa/pts/8 -bash1 ~ 19:08 0$
The dash seems to be coming from the '\s' bash(1) -> PROMPTING ->
backslash-escaped special characters:
2022-06-02 19:12:58 dpchrist@laalaa ~
$ PS1="\\s"
-bash
David
Re: Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
* David Christensen [22-06/02=Th 18:01 -0700]:
>On 6/2/22 17:12, Will Mengarini wrote:
>> * David Christensen [22-06/02=Th 15:50 -0700]:
>>> On 6/2/22 15:13, Will Mengarini wrote:
>
In this transcript, the number before the prompt-ending '$' is $?:
debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
Died at -e line 1.
debian/pts/4 bash3 ~ 14:57 2$
>>>
>>> What is your shell? PS1?
>>
>> The shell is Bash 5.1.4.
>> PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
>
> Interesting.
>
> This is my daily driver:
>
> 2022-06-02 17:38:55 dpchrist@laalaa ~
> $ cat /etc/debian_version ; uname -a ; dpkg-query -W bash
> 11.3
> Linux laalaa 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64
> GNU/Linux
> bash 5.1-2+b3
>
> This is my PS1. '\u' does not work on all of Debian, FreeBSD,
> Cygwin, and macOS, so the expansion of ${USER} is inserted
> between two string literals when .profile runs and sets PS1:
>
> 2022-06-02 17:39:09 dpchrist@laalaa ~
> $ grep PS1 .profile
> export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
> #export PS1='\n\D{%Y-%m-%d %H:%M:%S} \u@\h \w\n\$ '
>
> Testing your PS1:
>
> 2022-06-02 17:45:03 dpchrist@laalaa ~
> $ PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
> laalaa/ -bash1 ~ 17:45 0$
>
> The snippet '${TTY#/dev/}' seems to produce ' -' on my computer.
> How does your computer produce 'pts/4 ' and what does it mean?
'pts/4' is an abbreviation of '/dev/pts/4', pseudoterminal 4.
TTY is `tty`; it's been so long I'd forgotten that's not available in
all shells. You should have the 'tty' program; it's in coreutils.
> Is there a reason why you are using
> double quotes, rather than single quotes?
So I can interpolate stuff like ${TTY#/dev/}. In your case,
you'll need to set TTY=`tty` before setting PS1, so Bash
can use string substitution to remove '/dev/' from it.
Bash and the PS1 environment variable [was: grep: show matching line from pattern file]
On 6/2/22 17:12, Will Mengarini wrote:
> * David Christensen [22-06/02=Thu 15:50 -0700]:
>> On 6/2/22 15:13, Will Mengarini wrote:
>>> In this transcript, the number before the prompt-ending '$' is $?:
>>>
>>> debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
>>> debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
>>> Died at -e line 1.
>>> debian/pts/4 bash3 ~ 14:57 2$
>>>
>>
>> What is your shell? PS1?
>
> The shell is Bash 5.1.4.
> PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
Interesting.
This is my daily driver:
2022-06-02 17:38:55 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a ; dpkg-query -W bash
11.3
Linux laalaa 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29)
x86_64 GNU/Linux
bash5.1-2+b3
This is my PS1. '\u' does not work on all of Debian, FreeBSD, Cygwin,
and macOS, so the expansion of ${USER} is inserted between two string
literals when .profile runs and sets PS1:
2022-06-02 17:39:09 dpchrist@laalaa ~
$ grep PS1 .profile
export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '
#export PS1='\n\D{%Y-%m-%d %H:%M:%S} \u@\h \w\n\$ '
Testing your PS1:
2022-06-02 17:45:03 dpchrist@laalaa ~
$ PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
laalaa/ -bash1 ~ 17:45 0$
The snippet '${TTY#/dev/}' seems to produce ' -' on my computer. How
does your computer produce 'pts/4 ' and what does it mean?
Is there a reason why you are using double quotes, rather than single
quotes?
David
Re: grep: show matching line from pattern file
* David Christensen [22-06/02=Thu 15:50 -0700]:
> On 6/2/22 15:13, Will Mengarini wrote:
>> * Greg Wooledge [22-05/28=Sa 17:11 -0400]:
>>> [...]
>>> #!/usr/bin/perl
>>> use strict; use warnings;
>>> [...]
>>> open PATS, ">> [...]
>>
>> You need "or die", not "|| die", because of precedence: what you coded
>> checks whether "> wanted to check whether the result of open() is logically true.
>
> +1 That is a good explanation of a Perl fine point/ gotcha.
>
>> In this transcript, the number before the prompt-ending '$' is $?:
>>
>> debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die'
>> debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die'
>> Died at -e line 1.
>> debian/pts/4 bash3 ~ 14:57 2$
>>
>
> What is your shell? PS1?
The shell is Bash 5.1.4. My PS1 is constructed by an elaborate script
that's old enough to have sex in Thailand, but you can get the effect
of what I posted by setting PS1 with the line
PS1="\\h/${TTY#/dev/} \\s$SHLVL \\w \\A \$?\\\$"
assuming you're running at least Bash 2.05a. You may prefer
PS1="\\h/${TTY#/dev/} \\s^$SHLVL \\w \\A \$?\\\$"
.
My original script was coded for Bash 1.4.7, and had to do
PS1="\\h${TTY#/dev/} \\s$SHLVL \\w \`s=\$?;date +%H:%M;exit \$s\` \$?\\\$"
because \A wasn't available, so 'date' had to be run in a subshell
that needed to take care to save and restore $?. (The variable
it uses for that, s, goes away when the subshell does; and that
scary-looking exit just exits the subshell, resetting $?.)
--
Will Mengarini
Free software: the Source will be with you, always.
sh -c 'echo -n MENGARINI|sum -s|colrm 4'
Re: grep: show matching line from pattern file
On 6/2/22 15:13, Will Mengarini wrote: * Greg Wooledge [22-05/28=Sa 17:11 -0400]: [...] #!/usr/bin/perl use strict; use warnings; [...] open PATS, " You need "or die", not "|| die", because of precedence: what you coded checks whether " +1 That is a good explanation of a Perl fine point/ gotcha. In this transcript, the number before the prompt-ending '$' is $?: debian/pts/4 bash3 ~ 14:56 0$perl -e 'open "gweeblefleep" || die' debian/pts/4 bash3 ~ 14:57 0$perl -e 'open "gweeblefleep" or die' Died at -e line 1. debian/pts/4 bash3 ~ 14:57 2$ What is your shell? PS1? David
Re: grep: show matching line from pattern file
* Greg Wooledge [22-05/28=Sa 17:11 -0400]: > [...] > #!/usr/bin/perl > use strict; use warnings; > [...] > open PATS, " [...] You need "or die", not "|| die", because of precedence: what you coded checks whether " perl -le"print unpack '%C*',MENGARINI"
Re: grep: show matching line from pattern file
On Thu, Jun 02, 2022 at 03:12:23PM -0400, duh wrote: > > > Jim Popovitch wrote on 28/05/2022 21:40: > > > > I have a file of regex patterns and I use grep like so: > > > > > > > > ~$ grep -f patterns.txt /var/log/syslog > > > > > > > > What I'd like to get is a listing of all lines, specifically the line > > > > numbers of the regexps in patterns.txt, that match entries in > > > > /var/log/syslog. Is there a way to do this? > $cat -n /var/log/syslog | grep warn > > and it found "warn" in the syslog file and provided line numbers. I have > not used the -f option You're getting the line numbers from the log file. The OP wanted the line numbers of the patterns in the -f pattern file. Why? I have no idea. There is no standard option to do this, because it's not a common requirement. That's why I wrote one from scratch in perl.
Re: grep: show matching line from pattern file
On 5/29/22 9:44 AM, David Wright wrote: On Sun 29 May 2022 at 15:02:35 (+0200), Jörg-Volker Peetz wrote: Jim Popovitch wrote on 28/05/2022 21:40: Not exactly Debian specific, but hoping that someone here can help. I have a file of regex patterns and I use grep like so: ~$ grep -f patterns.txt /var/log/syslog What I'd like to get is a listing of all lines, specifically the line numbers of the regexps in patterns.txt, that match entries in /var/log/syslog. Is there a way to do this? How about this: $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt That will only work for literal patterns, not regex ones. Cheers, David. I may be missing a lot but I will risk throwing in my 2 cents (which is no longer worth the proverbial amount with the current inflation) $cat -n /var/log/syslog | grep warn and it found "warn" in the syslog file and provided line numbers. I have not used the -f option (but am now aware of it in the pastfrom your post -- thank you. Might file that away for future use but will not press my luck at the moment). My approach saves me the effort of writing a scriptl. Ahh, the Perl philosophy of "there is more than one way to do it".
Re: grep: show matching line from pattern file
On Sat, 2022-05-28 at 17:11 -0400, Greg Wooledge wrote:
> On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
> > On 2022-05-28 at 15:40, Jim Popovitch wrote:
> > > I have a file of regex patterns and I use grep like so:
> > >
> > > ~$ grep -f patterns.txt /var/log/syslog
> > >
> > > What I'd like to get is a listing of all lines, specifically the line
> > > numbers of the regexps in patterns.txt, that match entries in
> > > /var/log/syslog. Is there a way to do this?
> >
> > I don't know of a standardized way to do that (if anyone else wants to
> > suggest one, I'm open to learn), but of course it *can* be done, via
> > scripting. Off the top of my head, I came up with the following
> >
> > for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
> > if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
> > /dev/null ; then
> > echo $line ;
> > fi
> > done
>
> The quoting here is... completely absent (and that's extremely bad), but
> also importantly, one would ideally like to avoid running grep a thousand
> times, especially if the target logfile is large.
>
> I believe this is the kind of job for which perl is well-suited. I'm not
> great at perl, but I'll give it a shot.
>
> Here's a version with some extra information as output, so I can verify
> that it's doing something reasonably close to correct:
>
>
> #!/usr/bin/perl
> use strict; use warnings;
>
> my @patlist;
> open PATS, " chomp(@patlist = );
> close PATS;
>
> while (<>) {
> chomp;
> for (my $i = 0; $i <= $#patlist; $i++) {
> print "$i|$patlist[$i]|$_\n" if /$patlist[$i]/;
> }
> }
>
>
> Now, to test it, we need a patterns.txt file:
>
>
> unicorn:~$ cat patterns.txt
> PATH
> HOME|~
> a...e
>
>
> And an input (log) file:
>
>
> unicorn:~$ cat file
> zebra
> Home, home on the range.
> Oops, I meant HOME on the range.
>
> applesauce
>
>
> And here's what it does:
>
>
> unicorn:~$ ./foo file
> 1|HOME|~|Oops, I meant HOME on the range.
> 2|a...e|applesauce
>
>
> Pattern numbers 1 and 2 (the second and third, since it starts at 0) were
> matched, so we have a line for each of those.
>
> If that's kinda what you wanted, then you can adjust this to do precisely
> what you wanted. It shouldn't take a lot of work, I hope. Well, I guess
> that depends on what you really want.
>
> Bash is not well-suited to this task, and even if we were to take The
> Wanderer's script and fix all the issues in it, it would still be a
> vastly inferior solution. Some tools are just not meant for some jobs.
>
Thanks Greg, that is exactly what I needed, and double thanks for the
details in explaining it, etc.
-Jim P.
Re: grep: show matching line from pattern file
On Sun 29 May 2022 at 15:02:35 (+0200), Jörg-Volker Peetz wrote: > Jim Popovitch wrote on 28/05/2022 21:40: > > Not exactly Debian specific, but hoping that someone here can help. > > > > I have a file of regex patterns and I use grep like so: > > > > ~$ grep -f patterns.txt /var/log/syslog > > > > What I'd like to get is a listing of all lines, specifically the line > > numbers of the regexps in patterns.txt, that match entries in > > /var/log/syslog. Is there a way to do this? > > How about this: > > $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt That will only work for literal patterns, not regex ones. Cheers, David.
Re: grep: show matching line from pattern file
Jim Popovitch wrote on 28/05/2022 21:40: Not exactly Debian specific, but hoping that someone here can help. I have a file of regex patterns and I use grep like so: ~$ grep -f patterns.txt /var/log/syslog What I'd like to get is a listing of all lines, specifically the line numbers of the regexps in patterns.txt, that match entries in /var/log/syslog. Is there a way to do this? -Jim P. How about this: $ grep -of patterns.txt /var/log/syslog.1 | grep -n -f - patterns.txt Regards, Jörg.
Re: grep: show matching line from pattern file
On 2022-05-28 at 17:11, Greg Wooledge wrote: > On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote: > >> On 2022-05-28 at 15:40, Jim Popovitch wrote: >> > I have a file of regex patterns and I use grep like so: >> > >> >~$ grep -f patterns.txt /var/log/syslog >> > >> > What I'd like to get is a listing of all lines, specifically the line >> > numbers of the regexps in patterns.txt, that match entries in >> > /var/log/syslog. Is there a way to do this? >> >> I don't know of a standardized way to do that (if anyone else wants to >> suggest one, I'm open to learn), but of course it *can* be done, via >> scripting. Off the top of my head, I came up with the following >> >> for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do >> if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog > >> /dev/null ; then >> echo $line ; >> fi >> done > > The quoting here is... completely absent (and that's extremely bad), but > also importantly, one would ideally like to avoid running grep a thousand > times, especially if the target logfile is large. A brother of mine has schooled me on several things I did wrong here already, and I am now going over my long-tail stockpile of scripts with shellcheck, seeing what I can learn. (I normally do quote variables, but for some reason it slipped my mind this time until after I'd already hit Send. Some of these are known-safe values which won't need quoting, but others aren't, so the principle remains valid to cite.) This wasn't especially intended as a great solution, in any case; it started out as me trying to write shell-like pseudocode, but then I couldn't see any obvious reason why it wouldn't work, and when I tried it out I only needed a few tweaks before it did. There's a *reason* I had the "YMMV" on that message. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
Re: grep: show matching line from pattern file
On Sat, May 28, 2022 at 04:02:39PM -0400, The Wanderer wrote:
> On 2022-05-28 at 15:40, Jim Popovitch wrote:
> > I have a file of regex patterns and I use grep like so:
> >
> >~$ grep -f patterns.txt /var/log/syslog
> >
> > What I'd like to get is a listing of all lines, specifically the line
> > numbers of the regexps in patterns.txt, that match entries in
> > /var/log/syslog. Is there a way to do this?
>
> I don't know of a standardized way to do that (if anyone else wants to
> suggest one, I'm open to learn), but of course it *can* be done, via
> scripting. Off the top of my head, I came up with the following
>
> for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do
> if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog >
> /dev/null ; then
> echo $line ;
> fi
> done
The quoting here is... completely absent (and that's extremely bad), but
also importantly, one would ideally like to avoid running grep a thousand
times, especially if the target logfile is large.
I believe this is the kind of job for which perl is well-suited. I'm not
great at perl, but I'll give it a shot.
Here's a version with some extra information as output, so I can verify
that it's doing something reasonably close to correct:
#!/usr/bin/perl
use strict; use warnings;
my @patlist;
open PATS, ");
close PATS;
while (<>) {
chomp;
for (my $i = 0; $i <= $#patlist; $i++) {
print "$i|$patlist[$i]|$_\n" if /$patlist[$i]/;
}
}
Now, to test it, we need a patterns.txt file:
unicorn:~$ cat patterns.txt
PATH
HOME|~
a...e
And an input (log) file:
unicorn:~$ cat file
zebra
Home, home on the range.
Oops, I meant HOME on the range.
applesauce
And here's what it does:
unicorn:~$ ./foo file
1|HOME|~|Oops, I meant HOME on the range.
2|a...e|applesauce
Pattern numbers 1 and 2 (the second and third, since it starts at 0) were
matched, so we have a line for each of those.
If that's kinda what you wanted, then you can adjust this to do precisely
what you wanted. It shouldn't take a lot of work, I hope. Well, I guess
that depends on what you really want.
Bash is not well-suited to this task, and even if we were to take The
Wanderer's script and fix all the issues in it, it would still be a
vastly inferior solution. Some tools are just not meant for some jobs.
Re: grep: show matching line from pattern file
On 2022-05-28 at 15:40, Jim Popovitch wrote: > Not exactly Debian specific, but hoping that someone here can help. > > I have a file of regex patterns and I use grep like so: > >~$ grep -f patterns.txt /var/log/syslog > > What I'd like to get is a listing of all lines, specifically the line > numbers of the regexps in patterns.txt, that match entries in > /var/log/syslog. Is there a way to do this? I don't know of a standardized way to do that (if anyone else wants to suggest one, I'm open to learn), but of course it *can* be done, via scripting. Off the top of my head, I came up with the following for line in $(seq 1 $(wc -l patterns.txt | cut -d ' ' -f 1)) ; do if grep $(head -n $line patterns.txt | tail -n 1) /var/log/syslog > /dev/null ; then echo $line ; fi done I just tested that on my own system, with a different file (since I'm not root right now) and a couple of exact-string patterns found by examining that file, and it seems to work as intended. YMMV. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
grep: show matching line from pattern file
Not exactly Debian specific, but hoping that someone here can help. I have a file of regex patterns and I use grep like so: ~$ grep -f patterns.txt /var/log/syslog What I'd like to get is a listing of all lines, specifically the line numbers of the regexps in patterns.txt, that match entries in /var/log/syslog. Is there a way to do this? -Jim P.
