Re: [gentoo-user] is a nice place :-D
On Wed, May 18, 2011 at 10:28:55PM +0200, Alan McKinnon wrote The difference between bash and perl? Perl was inspired by a linguist, who at least puts his foot down at the truly crazy suggestions. Bash has no such thing. Perl is a bloated mediocre operating system, complete with repo (CPAN). But it lacks a lightweight extraction/reporting language. BTW, the most powerful little-known gem in bash is that you can open filehandles to do sequential reads and writes, just like in BASIC. For examples, see... http://blogmag.net/blog/read/48/File_descriptors_in_shell -- Walter Dnes waltd...@waltdnes.org
Re: [gentoo-user] is a nice place :-D
On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote: Not addressed at you, specifically, but it rather seems like sed awk are much under-appreciated these days. I'd guess that this may be due to the changing nature of *nix users, but they seem to have gone out of fashion. Aside from sed's simple replace, I have certainly never learned to do anything useful with them. They both have a steep initial learning curve, which leads to their adoption being put off. I put awk in the same category as screen, one of those programs that you hear people going on about for years, but always manage to put off trying them. Once you do try them, you use them for everything but slicing bread. -- Neil Bothwick Hors d'oeuvres: 3 sandwiches cut into 40 pieces. signature.asc Description: PGP signature
Re: [gentoo-user] is a nice place :-D
Apparently, though unproven, at 12:17 on Wednesday 18 May 2011, Neil Bothwick did opine thusly: On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote: Not addressed at you, specifically, but it rather seems like sed awk are much under-appreciated these days. I'd guess that this may be due to the changing nature of *nix users, but they seem to have gone out of fashion. Aside from sed's simple replace, I have certainly never learned to do anything useful with them. They both have a steep initial learning curve, which leads to their adoption being put off. I put awk in the same category as screen, one of those programs that you hear people going on about for years, but always manage to put off trying them. Once you do try them, you use them for everything but slicing bread. Add bash to that list. Have you read the full man page for the bloody thing? No wonder most folk stop at launching it after login -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] is a nice place :-D
On Wed, 18 May 2011 21:03:47 +0200, Alan McKinnon wrote: They both have a steep initial learning curve, which leads to their adoption being put off. I put awk in the same category as screen, one of those programs that you hear people going on about for years, but always manage to put off trying them. Once you do try them, you use them for everything but slicing bread. Add bash to that list. Have you read the full man page for the bloody thing? It's better than the zsh man page, which is split into several pages. That sounds like a good idea, and it would have been if there had been some sort of index so you knew which page to look on. -- Neil Bothwick Top Oxymorons Number 46: Found missing signature.asc Description: PGP signature
Re: [gentoo-user] is a nice place :-D
Hi, Alan. On Wed, May 18, 2011 at 09:03:47PM +0200, Alan McKinnon wrote: Apparently, though unproven, at 12:17 on Wednesday 18 May 2011, Neil Bothwick did opine thusly: On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote: Not addressed at you, specifically, but it rather seems like sed awk are much under-appreciated these days. I'd guess that this may be due to the changing nature of *nix users, but they seem to have gone out of fashion. Aside from sed's simple replace, I have certainly never learned to do anything useful with them. They both have a steep initial learning curve, which leads to their adoption being put off. I put awk in the same category as screen, one of those programs that you hear people going on about for years, but always manage to put off trying them. Once you do try them, you use them for everything but slicing bread. Add bash to that list. Have you read the full man page for the bloody thing? You're not meant to read that man page through from beginning to end. Anybody who could learn bash that way would be superhuman. Unfortunately, the info pages for bash are not well organised. So beginners have to learn from books, many of which are not good. And bash is about the most disorganised, arbitrary language around, full of crazy little quirks and odd sytaxes. And I love it. ;-) No wonder most folk stop at launching it after login -- alan dot mckinnon at gmail dot com -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] is a nice place :-D
Apparently, though unproven, at 22:15 on Wednesday 18 May 2011, Alan Mackenzie did opine thusly: Hi, Alan. On Wed, May 18, 2011 at 09:03:47PM +0200, Alan McKinnon wrote: Apparently, though unproven, at 12:17 on Wednesday 18 May 2011, Neil Bothwick did opine thusly: On Tue, 17 May 2011 18:38:33 +0100, Stroller wrote: Not addressed at you, specifically, but it rather seems like sed awk are much under-appreciated these days. I'd guess that this may be due to the changing nature of *nix users, but they seem to have gone out of fashion. Aside from sed's simple replace, I have certainly never learned to do anything useful with them. They both have a steep initial learning curve, which leads to their adoption being put off. I put awk in the same category as screen, one of those programs that you hear people going on about for years, but always manage to put off trying them. Once you do try them, you use them for everything but slicing bread. Add bash to that list. Have you read the full man page for the bloody thing? You're not meant to read that man page through from beginning to end. Um, I did Anybody who could learn bash that way would be superhuman. I doubt I learned much though. I even took the effort to reformat it as an OOo doc so I could find stuff and give it to others. It was an interesting exercise, not necessary an interesting *learning* exercise Unfortunately, the info pages for bash are not well organised. So beginners have to learn from books, many of which are not good. And bash is about the most disorganised, arbitrary language around, full of crazy little quirks and odd sytaxes. And I love it. ;-) The difference between bash and perl? Perl was inspired by a linguist, who at least puts his foot down at the truly crazy suggestions. Bash has no such thing. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] is a nice place :-D
On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip |
\ awk '{print $1}' | sort | uniq | wc
In true grand Unix tradition you cannot get quicker, dirtier or more
effective than that
awk does pattern matching, o you can ditch the grep stage and use
awk '! /myip/ {print $1}'
You could use awk to search for the GET patterns too, not only saving yet
another process, but making sure that no one else, including you next
month, can work out what the command is supposed to do.
sort -u would save having a separate process for uniq, but I've no idea
if it's faster. It's only worth using sort -u if you would use uniq with
no arguments.
--
Neil Bothwick
- We are but packets in the internet of Life-
signature.asc
Description: PGP signature
Re: [gentoo-user] is a nice place :-D
On 2011-05-17, Neil Bothwick n...@digimed.co.uk wrote:
On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip |
\ awk '{print $1}' | sort | uniq | wc
In true grand Unix tradition you cannot get quicker, dirtier or more
effective than that
awk does pattern matching, o you can ditch the grep stage and use
awk '! /myip/ {print $1}'
You could use awk to search for the GET patterns too, not only saving yet
another process, but making sure that no one else, including you next
month, can work out what the command is supposed to do.
Meh, me forgetting what an awk snippet do? Never!
sed ... now that's a wholly different story :-P
sort -u would save having a separate process for uniq, but I've no idea
if it's faster. It's only worth using sort -u if you would use uniq with
no arguments.
And you can actually do the 'uniq' or '-u' function within awk. Quite
easily, in fact.
Here's a sample of awk doing uniq:
awk '!x[$1]++ { print $1 }'
Benefit? It doesn't care if the non-unique lines are one-after-another
or spread all over the text. The above snippet prints only the first
occurence. Combine that with a test for match:
awk '!x[$1]++ $0 ~ /awesome_regex_pattern/ {print $1}'
then with a test for negated match
awk '!x[$1]++ $0 ~ /awesome_regex_pattern/ $0 !~
/more_awesome_regex/ {print $1}'
Rgds,
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/
Re: [gentoo-user] is a nice place :-D
On Tue, May 17, 2011 at 5:43 AM, Pandu Poluan pa...@poluan.info wrote:
On 2011-05-17, Neil Bothwick n...@digimed.co.uk wrote:
On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip |
\ awk '{print $1}' | sort | uniq | wc
In true grand Unix tradition you cannot get quicker, dirtier or more
effective than that
awk does pattern matching, o you can ditch the grep stage and use
awk '! /myip/ {print $1}'
You could use awk to search for the GET patterns too, not only saving yet
another process, but making sure that no one else, including you next
month, can work out what the command is supposed to do.
Meh, me forgetting what an awk snippet do? Never!
sed ... now that's a wholly different story :-P
sort -u would save having a separate process for uniq, but I've no idea
if it's faster. It's only worth using sort -u if you would use uniq with
no arguments.
And you can actually do the 'uniq' or '-u' function within awk. Quite
easily, in fact.
Here's a sample of awk doing uniq:
awk '!x[$1]++ { print $1 }'
Benefit? It doesn't care if the non-unique lines are one-after-another
or spread all over the text. The above snippet prints only the first
occurence. Combine that with a test for match:
awk '!x[$1]++ $0 ~ /awesome_regex_pattern/ {print $1}'
then with a test for negated match
awk '!x[$1]++ $0 ~ /awesome_regex_pattern/ $0 !~
/more_awesome_regex/ {print $1}'
Rgds,
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/
I have always wondered if there is a way to do awk '{ print $1}' using
only builtin bash functions when you only have a one line string
Re: [gentoo-user] is a nice place :-D
Juan Diego Tascón writes:
I have always wondered if there is a way to do awk '{ print $1}' using
only builtin bash functions when you only have a one line string
str=one two five
# remove all from the first blank on, but will not work with
# other whitespace
echo ${str%% *}
or
# set $1, $2, $3, ... to words of $str
set $str
echo $1
or
# create array holding one word per element
strarr=( $str )
echo $strarr (or echo ${strarr[0]})
Wonko
Re: [gentoo-user] is a nice place :-D
On Tue, May 17, 2011 at 8:36 AM, Alex Schuster wo...@wonkology.org wrote:
Juan Diego Tascón writes:
I have always wondered if there is a way to do awk '{ print $1}' using
only builtin bash functions when you only have a one line string
str=one two five
# remove all from the first blank on, but will not work with
# other whitespace
echo ${str%% *}
or
# set $1, $2, $3, ... to words of $str
set $str
echo $1
or
# create array holding one word per element
strarr=( $str )
echo $strarr (or echo ${strarr[0]})
Wonko
thanks for the info
Re: [gentoo-user] is a nice place :-D
On 2011-05-17, Alex Schuster wo...@wonkology.org wrote:
Juan Diego Tascón writes:
I have always wondered if there is a way to do awk '{ print $1}' using
only builtin bash functions when you only have a one line string
str=one two five
# remove all from the first blank on, but will not work with
# other whitespace
echo ${str%% *}
or
# set $1, $2, $3, ... to words of $str
set $str
echo $1
or
# create array holding one word per element
strarr=( $str )
echo $strarr (or echo ${strarr[0]})
Wonko
How about this:
str=one two three
read word etc $str
echo $word
(not tested, though)
--
Pandu E Poluan - IT Optimizer
My website: http://pandu.poluan.info/
Re: [gentoo-user] is a nice place :-D
Hello,
On Tue, 17 May 2011, Alan McKinnon wrote:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip | \
awk '{print $1}' | sort | uniq | wc
useless use of ...
awk '/GET \/Tmp\/Linux\/G/{ips[$1]++;}END{print length(ips);}' \
/var/log/apache2/access_log
I add each access to ips[IP] in case you'd want to print that to,
e.g. by using
END {
for( i in ips ) {
print i : ips[i] accesses;
}
print length(ips) unique IPs total;
}
as the END block.
HTH,
-dnh
--
Any research done on how to efficiently use computers has been long lost
in the mad rush to upgrade systems to do things that aren't needed by
people who don't understand what they are really supposed to do with
them. -- Graham Reed, in asr
Re: [gentoo-user] is a nice place :-D
On 17/5/2011, at 11:43am, Pandu Poluan wrote:
On 2011-05-17, Neil Bothwick n...@digimed.co.uk wrote:
On Tue, 17 May 2011 01:33:39 +0200, Alan McKinnon wrote:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip |
\ awk '{print $1}' | sort | uniq | wc
...
awk does pattern matching, o you can ditch the grep stage and use
awk '! /myip/ {print $1}'
...
Meh, me forgetting what an awk snippet do? Never!
sed ... now that's a wholly different story :-P
Not addressed at you, specifically, but it rather seems like sed awk are much
under-appreciated these days. I'd guess that this may be due to the changing
nature of *nix users, but they seem to have gone out of fashion. Aside from
sed's simple replace, I have certainly never learned to do anything useful with
them.
Stroller.
[gentoo-user] is a nice place :-D
After attempting to install for the first time last week, I started 3 different threads here looking for help. I'm pleased with the nature of the responses, and being able to succeed eventually using a mix of those responses and my own efforts digging into Google, gentoo.org and cranial cobwebs. So, thanks to all who replied, and even to those who showed interest without replying. For http://fm.no-ip.com/Tmp/Linux/G/, newly created to use with those three threads, 'cat /var/log/apache2/access_log | grep GET /Tmp/Linux/G | grep -v myip | sort outfile' generated 117 lines. That's a lot more hits than I can ever remember getting before when asking for help from a mailing list (even if it did take 5 days to accumulate so many). I'm curious if anyone here would like to offer a better variant of my local query that would limit the hit count so that no more than one hit per IP is represented in the output? My skill with such things is very limited. I can't think of the the name of a command to cut the IP off the front of each line, much less how to compare if it's a non-first instance to be discarded. Or, maybe there's an Apache utility for doing this that I just don't know about? -- The wise are known for their understanding, and pleasant words are persuasive. Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/
Re: [gentoo-user] is a nice place :-D
Apparently, though unproven, at 01:10 on Tuesday 17 May 2011, Felix Miata did
opine thusly:
After attempting to install for the first time last week, I started 3
different threads here looking for help. I'm pleased with the nature of the
responses, and being able to succeed eventually using a mix of those
responses and my own efforts digging into Google, gentoo.org and cranial
cobwebs. So, thanks to all who replied, and even to those who showed
interest without replying.
For http://fm.no-ip.com/Tmp/Linux/G/, newly created to use with those three
threads, 'cat /var/log/apache2/access_log | grep GET /Tmp/Linux/G | grep
-v myip | sort outfile' generated 117 lines. That's a lot more hits
than I can ever remember getting before when asking for help from a
mailing list (even if it did take 5 days to accumulate so many).
I'm curious if anyone here would like to offer a better variant of my local
query that would limit the hit count so that no more than one hit per IP is
represented in the output? My skill with such things is very limited. I
can't think of the the name of a command to cut the IP off the front of
each line, much less how to compare if it's a non-first instance to be
discarded. Or, maybe there's an Apache utility for doing this that I just
don't know about?
There's always a million ways to skin a cat like this. At a high volume site
you would of course not try and deal with this directly from the apache logs.
You would send them to syslog which would parse them and write them to a
database from where you could run sophisticated SQL.
There are also Apache analyser apps out there, google will find them.
But I think all that is overkill for what you want. Your command works fine
except for needing to discard duplicate IPs. You don't seem to need to know
the details of the GET, so just grab using awk the first field and sort | uniq
the result. It will run a tad quicker (and reveal less n00bness to your
audience) if you grep the file directly instead of cat | grep:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip | \
awk '{print $1}' | sort | uniq | wc
In true grand Unix tradition you cannot get quicker, dirtier or more effective
than that
--
alan dot mckinnon at gmail dot com
Re: [gentoo-user] is a nice place :-D
On Tue, May 17, 2011 at 01:33:39AM +0200, Alan McKinnon wrote:
grep GET /Tmp/Linux/G | /var/log/apache2/access_log | grep-v myip | \
awk '{print $1}' | sort | uniq | wc
In true grand Unix tradition you cannot get quicker, dirtier or more
effective
than that
You can replace sort | uniq by sort -u
And the Grand Unix Tradition probably would 'cut' instead of awk :)
While you are at it, an incantation that pipes grep to awk? Seriously?
W
--
Willie W. Wong ww...@math.princeton.edu
Data aequatione quotcunque fluentes quantitae involvente fluxiones invenire
et vice versa ~~~ I. Newton
