Re: tai64n -- why?

[Ben Beuchler]

On Sat, Aug 05, 2000 at 08:07:38PM -0700, Eric Cox wrote:

> But I'm getting back into it.  When I get it into a usable state I'll 
> post it on my website at www.ericcox.com.  Keep checking back...

Your mailstats program looks pretty cool.  Does it work with multilog
logs?  With tai64n and all that?

Ben

-- 
Ben Beuchler [EMAIL PROTECTED]
MAILER-DAEMON (612) 321-9290 x101
Bitstream Underground   www.bitstream.net

Re: tai64n -- why?

[Eric Cox]

Ben Beuchler wrote:
> 
> On Sat, Aug 05, 2000 at 11:54:41AM -0700, Eric Cox wrote:
> 
> > Have you looked into using libtai in your app?  It's Dan's library for
> > using tai, and could be linked with your application.  I currently
> > trying to write a Python wrapper for it so I can use it from my Python
> > scripts...
> 
> You're my hero!  The reason I don't use libtai is that I don't program
> in C.  I program in Python.  How much headway have you made?  Is it
> usable?

Heh, no way...  :(

But I'm getting back into it.  When I get it into a usable state I'll 
post it on my website at www.ericcox.com.  Keep checking back...

Eric

Re: tai64n -- why?

[Ben Beuchler]

On Sat, Aug 05, 2000 at 11:54:41AM -0700, Eric Cox wrote:

> Have you looked into using libtai in your app?  It's Dan's library for
> using tai, and could be linked with your application.  I currently
> trying to write a Python wrapper for it so I can use it from my Python
> scripts...

You're my hero!  The reason I don't use libtai is that I don't program
in C.  I program in Python.  How much headway have you made?  Is it
usable?

Gracias,
Ben

-- 
Ben Beuchler [EMAIL PROTECTED]
MAILER-DAEMON (612) 321-9290 x101
Bitstream Underground   www.bitstream.net

Re: tai64n -- why?

[Frank D. Cringle]

Ben Beuchler <[EMAIL PROTECTED]> writes:
> OK.  I can see where that is a strong argument for TAI64n.  I have tried
> to develop a way of converting from TAI64n to a human readable format
> but I'm afraid I'm having difficulty comprehending the format.  Has
> anyone written any perl/python code that translates to a human readable
> format?  Or can offer a brief "pseudo-code" algorithm?

Here is some stuff that I use in perl scripts.  There is plenty to
criticise here.  It assumes a 32-bit platform and fails after 2038.
It is much less efficient than tai64nlocal.  Handy if you just need
to output some lines that the script has selected, though.

# Simplistic tai64n converter.  Returns float time_t value or undef.
sub tai64n {
my(@x) = map { unpack "N", pack "H8", $_ }
$_[0] =~ /^\@?([0-9a-f]{8})([0-9a-f]{8})([0-9a-f]{8})/;
return unless @x == 3 and $x[0] == 0x4000;
$x[1]+$x[2]/1_000_000_000;
}

# replace tai64n timestamp at beginning of line and print result
sub putline {
my($stamp,$msg) = $_[0] =~ /^(\S+) (.*)/;
$stamp = tai64n($stamp);
my $int = int($stamp);
my $frac = int(($stamp-$int)*1_000_000_000);
my @t = localtime $int;
printf("%d-%02d-%02d %02d:%02d:%02d.%09d %s\n",
   $t[5]+1900,$t[4]+1,@t[3,2,1,0],$frac,$msg);
}

-- 
Frank Cringle,  [EMAIL PROTECTED]
voice: (+49 2304) 467101; fax: 943357

Re: tai64n -- why?

[Eric Cox]

Ben Beuchler wrote:
> 
> On Fri, Aug 04, 2000 at 07:11:52PM -0700, Russ Allbery wrote:
> 
> > syslog timestamps are amazingly annoying to try to parse.  TAI64 is
> > trivial to parse.  This is a significant improvement.
> >
> > ISO date/time format would also have been easy to parse, and I would have
> > been slightly happier with that, but TAI64 is definitely a *huge*
> > improvement over syslog if you want to do anything at all automated with
> > the logs.
> 
> OK.  I can see where that is a strong argument for TAI64n.  I have tried
> to develop a way of converting from TAI64n to a human readable format
> but I'm afraid I'm having difficulty comprehending the format.  Has
> anyone written any perl/python code that translates to a human readable
> format?  Or can offer a brief "pseudo-code" algorithm?
> 
> Or even a more clear description of the format would be helpful.  Dan's
> writeup left me a bit lost...

Have you looked into using libtai in your app?  It's Dan's library for 
using tai, and could be linked with your application.  I currently 
trying to write a Python wrapper for it so I can use it from my Python 
scripts...

Eric

Re: tai64n -- why?

[Ben Beuchler]

On Fri, Aug 04, 2000 at 07:11:52PM -0700, Russ Allbery wrote:

> syslog timestamps are amazingly annoying to try to parse.  TAI64 is
> trivial to parse.  This is a significant improvement.
> 
> ISO date/time format would also have been easy to parse, and I would have
> been slightly happier with that, but TAI64 is definitely a *huge*
> improvement over syslog if you want to do anything at all automated with
> the logs.

OK.  I can see where that is a strong argument for TAI64n.  I have tried
to develop a way of converting from TAI64n to a human readable format
but I'm afraid I'm having difficulty comprehending the format.  Has
anyone written any perl/python code that translates to a human readable
format?  Or can offer a brief "pseudo-code" algorithm?

Or even a more clear description of the format would be helpful.  Dan's
writeup left me a bit lost...

Gracias,
Ben

-- 
Ben Beuchler [EMAIL PROTECTED]
MAILER-DAEMON (612) 321-9290 x101
Bitstream Underground   www.bitstream.net

Re: tai64n -- why?

[Russ Allbery]

David Dyer-Bennet <[EMAIL PROTECTED]> writes:

> Yes, when I first looked at it.  As is often the case with Dan, I just
> disagree.  It's not straight text in the sense I mean; it's not human
> readable.  Of all the strange choices Dan's made that I've encountered
> in working with qmail, this is the first one that I fail completely to
> understand.  All the others, I see the tradeoffs and I see why he chose
> as he did, even if I might have chosen otherwise.  This one makes zero
> sense.  It's non-functional.  It doesn't connect to the way I work.

syslog timestamps are amazingly annoying to try to parse.  TAI64 is
trivial to parse.  This is a significant improvement.

ISO date/time format would also have been easy to parse, and I would have
been slightly happier with that, but TAI64 is definitely a *huge*
improvement over syslog if you want to do anything at all automated with
the logs.

-- 
Russ Allbery ([EMAIL PROTECTED]) 

Re: tai64n -- why?

[Michael T. Babcock]

Which part of "more like" did you miss?

Fixing seconds precision to anything is unnecessary ... with a decimal to
separate the fractional portion and a space to end it, it can be as short or
long as you wish -- a parser does not need that number to have 0 padding at
the end ... comment irrelevant.

The bracketed GMT+x is for the sake of comparing times made by systems that
don't have a friendly timezone system for conversion.  That said, preference
would be to store times in GMT and not need the spec ... (GMT+00) ... but
that's not as easy as it sounds sometimes, so the option was left.

Yes, I thought of those.

And yes, its sortable -- within a timezone.

Petr Novotny wrote:

> > Are you asking more for something like:
> >
> > 2000/07/31 06:02:10.42 (GMT+05)
> >
> > This has always been the date format I've prefered ... its sortable
> > (as the year comes first -- although its quite narrow-minded of me to
> > not allow for 5 digit dates),
>
> It's not sortable:
> 1. '+' precedes '-' in ASCII. Yuck.
> 2. How do you compare (cutting unimportant stuff) 17:30 (GMT+1)
> and 18:00 (GMT+0)? Go figure.
> 3. Fixing seconds precision to two decimals is wrong, too.

Re: tai64n -- why?

[Petr Novotny]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 1 Aug 00, at 9:58, Michael T. Babcock wrote:

> Are you asking more for something like:
> 
> 2000/07/31 06:02:10.42 (GMT+05)
> 
> This has always been the date format I've prefered ... its sortable
> (as the year comes first -- although its quite narrow-minded of me to
> not allow for 5 digit dates),

It's not sortable:
1. '+' precedes '-' in ASCII. Yuck.
2. How do you compare (cutting unimportant stuff) 17:30 (GMT+1) 
and 18:00 (GMT+0)? Go figure.
3. Fixing seconds precision to two decimals is wrong, too.

There goes.

> > You mean this:
> >Jul 31 06:02:10 gw qmail: 965023330.820010 status: local 1/10
> >remote 0/50
> > ?

What? You're timestamping your logs both by syslog and some 
timestamper from old daemontools, whatever-was-its-name, 
accustamp? Why on Earth? Either you want to log through syslogd 
or something which adds the stamps (you call them human 
readable), or through accustamp or multilog which adds tai or tai64 
timestamps.

And, for what it's worth, _my_ logs should be machine readable, 
not human readable. I inspect qmail logs only seldom, and I can 
surely pipe them through tai(64n)local; however, my log analyzing 
tools run much more often, and I expect lots of reliability from 
them. And I don't want to spend several hours explaining the log 
parser that on my system, the timestamp looks like 
"%D %d %M %h:%m:%.4s %TZ" or whatever.

-BEGIN PGP SIGNATURE-
Version: PGP 6.0.2 -- QDPGP 2.60 
Comment: http://community.wow.net/grt/qdpgp.html

iQA/AwUBOYbMfVMwP8g7qbw/EQICmACeLP7qwvTtHsOz9Exks6t+8YaF2u0AoOzr
YJ5FgM9B3kdr8pUKnRZk3s0O
=1OBP
-END PGP SIGNATURE-
--
Petr Novotny, ANTEK CS
[EMAIL PROTECTED]
http://www.antek.cz
PGP key ID: 0x3BA9BC3F
-- Don't you know there ain't no devil there's just God when he's drunk.
 [Tom Waits]

Re: tai64n -- why?

[Michael T. Babcock]

Are you asking more for something like:

2000/07/31 06:02:10.42 (GMT+05)

This has always been the date format I've prefered ... its sortable (as the year
comes first -- although its quite narrow-minded of me to not allow for 5 digit
dates), its human-readable, and quite parsable.  The decimal portion after the time
allows for sub-second time measurement.  I'm pretty sure adding a function to your
copy of multilog/whatever that converted from the tai64n format to something more
like the above wouldn't be that hard.

David Dyer-Bennet wrote:

>  > Would you prefer the splogger format (to wit, Unix timestamp with
>  > fractional part) instead? I'd do anything to use a logging format
>  > that avoids timezone dependency, and multilog/tai64n seems to do
>  > that job well.
>
> You mean this:
>Jul 31 06:02:10 gw qmail: 965023330.820010 status: local 1/10 remote 0/50
> ?
>
> It's better than tai64n, because syslog puts a real timestamp on, but
> that big chunk of meaningless numbers in the middle wastes a lot of
> the line and adds no useful information.  It's what I'm using now on
> my main server, but it's quite wasteful and annoying.  (But
> qmailanalog expects it)

Re: tai64n -- why?

[David Dyer-Bennet]

Chris, the Young One <[EMAIL PROTECTED]> writes on 1 August 2000 at 12:56:46 +1200
 > On Mon, Jul 31, 2000 at 12:54:23PM -0500, David Dyer-Bennet wrote:
 > !  This one
 > ! makes zero sense.  It's non-functional.  It doesn't connect to the way
 > ! I work. 
 > 
 > Would you prefer the splogger format (to wit, Unix timestamp with
 > fractional part) instead? I'd do anything to use a logging format
 > that avoids timezone dependency, and multilog/tai64n seems to do
 > that job well.

You mean this:
   Jul 31 06:02:10 gw qmail: 965023330.820010 status: local 1/10 remote 0/50
?

It's better than tai64n, because syslog puts a real timestamp on, but
that big chunk of meaningless numbers in the middle wastes a lot of
the line and adds no useful information.  It's what I'm using now on
my main server, but it's quite wasteful and annoying.  (But
qmailanalog expects it)

I can see the desire to have a timezone independent format if you're
reading logs from systems in multiple timezones.  I'm not.

Having anything other than my current timezone in plain ASCII is a big
lose for me; it means I can't correlate the logs either to each other,
or to the real world.  (Yes, the random numbers produced by tai
correlate to each other, but I can't remember them, whereas I can
remember that something happened in midafternoon pretty easily).

Multilog and splogger should really have selectable time format and
timezone settings.  I'd love to use multilog and tell it to use plain
text and central time, rather than having to involve post-processors
(which are a pain for the current log; for older logs it could be done
automatically at rollover).  You, on the other hand, could tell your
multilog to use GMT on all machines so that when you correlate them
across timezones they'd all match up right.  And then we'd both be
happier than we are now.
-- 
Photos: http://dd-b.lighthunters.net/ Minicon: http://www.mnstf.org/minicon
Bookworms: http://ouroboros.demesne.com/ SF: http://www.dd-b.net/dd-b 
David Dyer-Bennet / Welcome to the future! / [EMAIL PROTECTED]

Re: tai64n -- why?

[Chris, the Young One]

On Mon, Jul 31, 2000 at 12:54:23PM -0500, David Dyer-Bennet wrote:
!  This one
! makes zero sense.  It's non-functional.  It doesn't connect to the way
! I work. 

Would you prefer the splogger format (to wit, Unix timestamp with
fractional part) instead? I'd do anything to use a logging format
that avoids timezone dependency, and multilog/tai64n seems to do
that job well.

---Chris K.
-- 
 Chris, the Young One |_ Never brag about how your machines haven't been 
  Auckland, New Zealand |_ hacked, or your code hasn't been broken. It's 
http://cloud9.hedgee.com/ |_ guaranteed to bring the wrong kind of 
 PGP: 0xCCC6114E/0x706A6AAD |_ attention. ---Neil Schneider 

Re: tai64n -- why?

[Ben Beuchler]

On Mon, Jul 31, 2000 at 12:54:23PM -0500, David Dyer-Bennet wrote:

> Yes, when I first looked at it.  As is often the case with Dan, I just
> disagree.  It's not straight text in the sense I mean; it's not human
> readable.  Of all the strange choices Dan's made that I've encountered
> in working with qmail, this is the first one that I fail completely to
> understand.  All the others, I see the tradeoffs and I see why he
> chose as he did, even if I might have chosen otherwise.  This one
> makes zero sense.  It's non-functional.  It doesn't connect to the way
> I work. 

You have expressed my sentiments precisely.  It has left me sad and
confused.

Ben

-- 
Ben Beuchler [EMAIL PROTECTED]
MAILER-DAEMON (612) 321-9290 x101
Bitstream Underground   www.bitstream.net

Re: tai64n -- why?

[David Dyer-Bennet]

[EMAIL PROTECTED] <[EMAIL PROTECTED]> writes on 31 July 2000 at 10:50:23 -0700
 > On Mon, Jul 31, 2000 at 12:23:38PM -0500, David Dyer-Bennet wrote:
 > > Charles Cazabon <[EMAIL PROTECTED]> writes on 31 July 2000 at 11:20:48 
 >-0600
 > >  > David Dyer-Bennet <[EMAIL PROTECTED]> wrote:
 > >  > >  > 
 > >  > >  > Really? If I want to tail a log file, eg, I go like this:
 > >  > >  > 
 > >  > >  > tail ../someservice/current | tai64nlocal
 > >  > >  > 
 > >  > >  > and it all looks fine for humans.
 > >  >  
 > >  > > Yeah, it works fine for people who check log files by tailing them.  I
 > >  > > check them by bringing them into an emacs buffer, so the funny
 > >  > > timestamps make them darned near useless.
 > >  > 
 > >  > So why not tail them to a temp file and use emacs to view the temp file?
 > >  > Or write an emacs-lisp function to convert the timestamps.
 > > 
 > > If I'm going to go to effort to make it work the way I want, I think
 > > I'll just change multilog to use a sensible format.  It's silly having
 > > archival log files sitting there that don't mean anything without a
 > > conversion program; straight text is the appropriate format for log
 > > files. 
 > 
 > But it *is* straight text. The point about tai is that it's entirely
 > appropriate for log files that may live for a long time. Have you
 > read the rationale for tai at all?

Yes, when I first looked at it.  As is often the case with Dan, I just
disagree.  It's not straight text in the sense I mean; it's not human
readable.  Of all the strange choices Dan's made that I've encountered
in working with qmail, this is the first one that I fail completely to
understand.  All the others, I see the tradeoffs and I see why he
chose as he did, even if I might have chosen otherwise.  This one
makes zero sense.  It's non-functional.  It doesn't connect to the way
I work. 
-- 
Photos: http://dd-b.lighthunters.net/ Minicon: http://www.mnstf.org/minicon
Bookworms: http://ouroboros.demesne.com/ SF: http://www.dd-b.net/dd-b 
David Dyer-Bennet / Welcome to the future! / [EMAIL PROTECTED]

Re: tai64n -- why?

[markd]

On Mon, Jul 31, 2000 at 12:23:38PM -0500, David Dyer-Bennet wrote:
> Charles Cazabon <[EMAIL PROTECTED]> writes on 31 July 2000 at 11:20:48 -0600
>  > David Dyer-Bennet <[EMAIL PROTECTED]> wrote:
>  > >  > 
>  > >  > Really? If I want to tail a log file, eg, I go like this:
>  > >  > 
>  > >  > tail ../someservice/current | tai64nlocal
>  > >  > 
>  > >  > and it all looks fine for humans.
>  >  
>  > > Yeah, it works fine for people who check log files by tailing them.  I
>  > > check them by bringing them into an emacs buffer, so the funny
>  > > timestamps make them darned near useless.
>  > 
>  > So why not tail them to a temp file and use emacs to view the temp file?
>  > Or write an emacs-lisp function to convert the timestamps.
> 
> If I'm going to go to effort to make it work the way I want, I think
> I'll just change multilog to use a sensible format.  It's silly having
> archival log files sitting there that don't mean anything without a
> conversion program; straight text is the appropriate format for log
> files. 

But it *is* straight text. The point about tai is that it's entirely
appropriate for log files that may live for a long time. Have you
read the rationale for tai at all?


Regards.

Re: tai64n -- why?

[tony]

Isn't there something on this list about "profile not speculate"?

(defun convert-tai64nlocal (arg)
"generate a local, human timestamped buffer from a tai64 timestamped buffer"
  (interactive "p")
  (mark-whole-buffer)
  (shell-command-on-region
  (region-beginning) (region-end) "tai64nlocal" nil nil)
  )

works pretty fast, only issue it opens an output buffer, but OTOH, you
don't have to go to any effort ;^) 

You could also just take out the 't' in your "...log/run" files, no time
stamp, no problem. 

Regards,
Tony

On Mon, 31 Jul 2000, David Dyer-Bennet wrote:

> Charles Cazabon <[EMAIL PROTECTED]> writes on 31 July 2000 at 11:20:48 -0600
>  > David Dyer-Bennet <[EMAIL PROTECTED]> wrote:
>  > >  > 
>  > >  > Really? If I want to tail a log file, eg, I go like this:
>  > >  > 
>  > >  > tail ../someservice/current | tai64nlocal
>  > >  > 
>  > >  > and it all looks fine for humans.
>  >  
>  > > Yeah, it works fine for people who check log files by tailing them.  I
>  > > check them by bringing them into an emacs buffer, so the funny
>  > > timestamps make them darned near useless.
>  > 
>  > So why not tail them to a temp file and use emacs to view the temp file?
>  > Or write an emacs-lisp function to convert the timestamps.
> 
> If I'm going to go to effort to make it work the way I want, I think
> I'll just change multilog to use a sensible format.  It's silly having
> archival log files sitting there that don't mean anything without a
> conversion program; straight text is the appropriate format for log
> files. 
> 
> I had thought about writing a mode for TAI stamped log files that
> converts the timestamps, but that will be pretty slow since it'll have
> to change every line of the file.
> 

-- 
Tony Hansmann ([EMAIL PROTECTED])
Director of Technical Services
Quepasa.com, INC.
602-716-0100

Re: tai64n -- why?

[David Dyer-Bennet]

Charles Cazabon <[EMAIL PROTECTED]> writes on 31 July 2000 at 11:20:48 -0600
 > David Dyer-Bennet <[EMAIL PROTECTED]> wrote:
 > >  > 
 > >  > Really? If I want to tail a log file, eg, I go like this:
 > >  > 
 > >  > tail ../someservice/current | tai64nlocal
 > >  > 
 > >  > and it all looks fine for humans.
 >  
 > > Yeah, it works fine for people who check log files by tailing them.  I
 > > check them by bringing them into an emacs buffer, so the funny
 > > timestamps make them darned near useless.
 > 
 > So why not tail them to a temp file and use emacs to view the temp file?
 > Or write an emacs-lisp function to convert the timestamps.

If I'm going to go to effort to make it work the way I want, I think
I'll just change multilog to use a sensible format.  It's silly having
archival log files sitting there that don't mean anything without a
conversion program; straight text is the appropriate format for log
files. 

I had thought about writing a mode for TAI stamped log files that
converts the timestamps, but that will be pretty slow since it'll have
to change every line of the file.
-- 
Photos: http://dd-b.lighthunters.net/ Minicon: http://www.mnstf.org/minicon
Bookworms: http://ouroboros.demesne.com/ SF: http://www.dd-b.net/dd-b 
David Dyer-Bennet / Welcome to the future! / [EMAIL PROTECTED]

Re: tai64n -- why?

[Charles Cazabon]

David Dyer-Bennet <[EMAIL PROTECTED]> wrote:
>  > 
>  > Really? If I want to tail a log file, eg, I go like this:
>  > 
>  > tail ../someservice/current | tai64nlocal
>  > 
>  > and it all looks fine for humans.
 
> Yeah, it works fine for people who check log files by tailing them.  I
> check them by bringing them into an emacs buffer, so the funny
> timestamps make them darned near useless.

So why not tail them to a temp file and use emacs to view the temp file?
Or write an emacs-lisp function to convert the timestamps.

Charles 
-- 
---
Charles Cazabon<[EMAIL PROTECTED]>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
Any opinions expressed are just that -- my opinions.
---

Re: tai64n -- why?

[David Dyer-Bennet]

[EMAIL PROTECTED] <[EMAIL PROTECTED]> writes on 30 July 2000 at 21:21:19 -0700
 > On Sun, Jul 30, 2000 at 11:06:38PM -0500, Ben Beuchler wrote:
 > > I understand from DJB's website that TAI is a better way to deal with
 > > time functions than the typical unix localtime().  However, it seems to
 > > make a lot of things really awkward when it is used as the time stamp in
 > > a log file.
 > 
 > Really? If I want to tail a log file, eg, I go like this:
 > 
 > tail ../someservice/current | tai64nlocal
 > 
 > and it all looks fine for humans.

Yeah, it works fine for people who check log files by tailing them.  I
check them by bringing them into an emacs buffer, so the funny
timestamps make them darned near useless.

I actually find second resolution entirely adequate for log entries;
the great detail TAI provides is just wasted space to me, and
displaces other, more useful, information from the screen.
-- 
Photos: http://dd-b.lighthunters.net/ Minicon: http://www.mnstf.org/minicon
Bookworms: http://ouroboros.demesne.com/ SF: http://www.dd-b.net/dd-b 
David Dyer-Bennet / Welcome to the future! / [EMAIL PROTECTED]

Re: tai64n -- why?

[markd]

On Sun, Jul 30, 2000 at 11:06:38PM -0500, Ben Beuchler wrote:
> I understand from DJB's website that TAI is a better way to deal with
> time functions than the typical unix localtime().  However, it seems to
> make a lot of things really awkward when it is used as the time stamp in
> a log file.

Really? If I want to tail a log file, eg, I go like this:

tail ../someservice/current | tai64nlocal

and it all looks fine for humans.

> Any particular idea why DJB chose to use it for output in multilog?  It
> seems to me that all it accomplishes is adding extra steps to gleaning
> any useful info from the logs...

Well, there are two issues here. One is the use of that particular
form of timestamp and the granularity of the timestamp. The granularity
of syslog is a second which is no where near good enough for timing
events that occur at sub-second rates. Knowing the author of syslog, I
can guess why he thought that a second was plenty accurate enough for
his MTA. But in the real world of modern computing, events happen
more briefly.

Once you accept that the current mainstream logging system isn't
giving sufficiently useful timestamps, then any change is going to
carry some level of awkwardness. What DjB has done is pick a timestamp
that has some pretty useful characteristics. Importantly, he has chosen
one which is as close to absolute as is possible and one which is
easy to use with date arithmetic.


Regards.