Re: Best tool to open & search large log files (5GB+)

2016-11-19 Thread Dave 
At work I mostly use Splunk, but then, I don't have to pay for it. At home, 
and sometimes at work if I have the raw logs available, I use grep to chop 
up the files into manageable chunks based on what I'm looking for. I don't 
mean the weak, out-of-date BSD grep Apple ships with Xcode. I get the 
latest version of gnu grep and build it with the latest PCRE, and not just 
because I find PCRE indispensible for this kind of work, but also because 
gnu grep is wicked fast.

Usually I can exctract what I want in one pass and redirect it to a file, 
but occasionally I pipe it back to grep or run it through sed (also the 
latest gnu version) and then to disk. This way you get smaller, more 
manageable files you can open in BBEdit or whatever and and analyze 
further. You can even get basic statistics like browser share by grepping 
specific patterns in User-Agent strings into separate files and counting 
the lines with wc -l. (If you don't need to keep the files you can just 
pipe grep to wc.)

awk is another tool people find useful, but I'm not that proficient with 
it. I don't know if I'm less proficient because I've found it less useful 
or the other way around, but the thing I love about Unix is that it has 
this amazing programming environment filled with useful, interoperable 
tools that let you accomplish tasks in all sorts of different ways. Most 
people probably never learn more than a few of them, and which ones they 
latch onto is either a historical happenstance or because different people 
just think differently and approach problems idiosyncratically.

On Thursday, November 17, 2016 at 12:06:53 PM UTC-5, Jonathan Duke wrote:
>
> I love BBEdit (since the classic days), but when I'm trying to open and 
> search some immense log files from our server it just isn't doing the job.
>
> I know it isn't the best tool for something like this, but when all you 
> have is a hammer…
>
> What would people suggest for files of this size (I'm downloading a log 
> file right now that is already 14 GB of an unknown total).
>
> Thanks.
>
> Cheers,
> Jon
>

-- 
This is the BBEdit Talk public discussion group. If you have a 
feature request or would like to report a problem, please email
"supp...@barebones.com" rather than posting to the group.
Follow @bbedit on Twitter: 
--- 
You received this message because you are subscribed to the Google Groups 
"BBEdit Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bbedit+unsubscr...@googlegroups.com.
To post to this group, send email to bbedit@googlegroups.com.
Visit this group at https://groups.google.com/group/bbedit.

Re: Best tool to open & search large log files (5GB+)

2016-11-17 Thread Fletcher Sandbeck 
The size of the log files is why there are specialty programs to summarize and 
analyze them. However, I usually use a combination of CLI tools and BBEdit to 
pull out the information that I want. Some of the CLI tools seem more efficient 
than loading the file into BBEdit and using its tools to accomplish these same 
things.

For example, I wanted to pull 404 errors out of a folder of tab-delimited log 
files so I used grep to filter out lines which contained a 404 field and piped 
that to cut which pulled out just the 8th field (which was the file name in 
this log), sorted the results, filtered to unique, and piped to BBEdit for 
display and further processing.

grep "\t404\t" * | cut -f 8 | sort | uniq | bbedit

The actual command I ended up with did two passes to make sure I didn't have 
any records which contained a length field of 404 or the like. The cut could be 
first, but the two passes seemed faster since the grep filtered out most of the 
200 and 30x entries.

grep "\t404\t" * | cut -f 8,9 | grep "\t404" | cut -f 1 | sort | uniq | bbedit

Hope this helps,

[fletcher]


> On Nov 17, 2016, at 7:40 AM, Jonathan Duke  wrote:
> 
> I love BBEdit (since the classic days), but when I'm trying to open and 
> search some immense log files from our server it just isn't doing the job.
> 
> I know it isn't the best tool for something like this, but when all you have 
> is a hammer…
> 
> What would people suggest for files of this size (I'm downloading a log file 
> right now that is already 14 GB of an unknown total).
> 
> Thanks.
> 
> Cheers,
> Jon
> 
> -- 
> This is the BBEdit Talk public discussion group. If you have a 
> feature request or would like to report a problem, please email
> "supp...@barebones.com" rather than posting to the group.
> Follow @bbedit on Twitter: 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "BBEdit Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to bbedit+unsubscr...@googlegroups.com.
> To post to this group, send email to bbedit@googlegroups.com.
> Visit this group at https://groups.google.com/group/bbedit.

-- 
This is the BBEdit Talk public discussion group. If you have a 
feature request or would like to report a problem, please email
"supp...@barebones.com" rather than posting to the group.
Follow @bbedit on Twitter: 
--- 
You received this message because you are subscribed to the Google Groups 
"BBEdit Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bbedit+unsubscr...@googlegroups.com.
To post to this group, send email to bbedit@googlegroups.com.
Visit this group at https://groups.google.com/group/bbedit.

Re: Best tool to open & search large log files (5GB+)

2016-11-17 Thread Kyle DeMilo 
Not a server admin, but why isn't the server segmenting that log file?
On Thu, Nov 17, 2016 at 12:06 PM Jonathan Duke  wrote:

> I love BBEdit (since the classic days), but when I'm trying to open and
> search some immense log files from our server it just isn't doing the job.
>
> I know it isn't the best tool for something like this, but when all you
> have is a hammer…
>
> What would people suggest for files of this size (I'm downloading a log
> file right now that is already 14 GB of an unknown total).
>
> Thanks.
>
> Cheers,
> Jon
>
> --
> This is the BBEdit Talk public discussion group. If you have a
> feature request or would like to report a problem, please email
> "supp...@barebones.com" rather than posting to the group.
> Follow @bbedit on Twitter: 
> ---
> You received this message because you are subscribed to the Google Groups
> "BBEdit Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbedit+unsubscr...@googlegroups.com.
> To post to this group, send email to bbedit@googlegroups.com.
> Visit this group at https://groups.google.com/group/bbedit.
>

-- 
This is the BBEdit Talk public discussion group. If you have a 
feature request or would like to report a problem, please email
"supp...@barebones.com" rather than posting to the group.
Follow @bbedit on Twitter: 
--- 
You received this message because you are subscribed to the Google Groups 
"BBEdit Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bbedit+unsubscr...@googlegroups.com.
To post to this group, send email to bbedit@googlegroups.com.
Visit this group at https://groups.google.com/group/bbedit.

Best tool to open & search large log files (5GB+)

2016-11-17 Thread Jonathan Duke 
I love BBEdit (since the classic days), but when I'm trying to open and 
search some immense log files from our server it just isn't doing the job.

I know it isn't the best tool for something like this, but when all you 
have is a hammer…

What would people suggest for files of this size (I'm downloading a log 
file right now that is already 14 GB of an unknown total).

Thanks.

Cheers,
Jon

-- 
This is the BBEdit Talk public discussion group. If you have a 
feature request or would like to report a problem, please email
"supp...@barebones.com" rather than posting to the group.
Follow @bbedit on Twitter: 
--- 
You received this message because you are subscribed to the Google Groups 
"BBEdit Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bbedit+unsubscr...@googlegroups.com.
To post to this group, send email to bbedit@googlegroups.com.
Visit this group at https://groups.google.com/group/bbedit.