Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 28 sept. 2011, at 13:38, Paul Linehan wrote: >> The granularity I'm looking for is between 1 second and 10 seconds. Cron is >> not >> an option here. > > I woke up this morning and there is a way that cron *_could_* do what you > want. You appear to have figured out a way that suits you, but

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011/9/27 Patrick Proniewski : >> I can't see why you would want to do this more than once every minute >> - or do you? > The granularity I'm looking for is between 1 second and 10 seconds. Cron is > not > an option here. I woke up this morning and there is a way that cron

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

Am 27.09.2011 23:07, schrieb Patrick Proniewski: > On 27 sept. 2011, at 20:18, Gabor Grothendieck wrote: > >> gawk has fflush() > > > On 27 sept. 2011, at 20:29, Roger Andersson wrote: > >> stdbuf? unbuffer? > > > none of them is available out of the box on Mac OS X, or FreeBSD. > gawk can

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 23:11, Scott Hess wrote: > On Tue, Sep 27, 2011 at 2:07 PM, Patrick Proniewski wrote: > >> On 27 sept. 2011, at 20:18, Gabor Grothendieck wrote: >>> gawk has fflush() >> >> none of them is available out of the box on Mac OS X, or FreeBSD. gawk can >> be

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 20:44, Paul Linehan wrote: > 2011/9/27 Patrick Proniewski : > >>> Take a look at a utility called dstat. > >> no, it's linux only. > > But it is written in Python - so it should be relatively > transportable. and it relies on /proc/, Mac OS X does not

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 20:41, Paul Linehan wrote: > 2011/9/27 Patrick Proniewski : > >> That's what I do, but I think using a loop is ugly, and I would like to find >> a way >> to feed data continuously into sqlite. > > I can't see why you would want to do this more than once

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On Tue, Sep 27, 2011 at 2:07 PM, Patrick Proniewski wrote: > On 27 sept. 2011, at 20:18, Gabor Grothendieck wrote: > > gawk has fflush() > > none of them is available out of the box on Mac OS X, or FreeBSD. gawk can > be installed, but I'd rather use my "while true" loop

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 20:18, Gabor Grothendieck wrote: > gawk has fflush() On 27 sept. 2011, at 20:29, Roger Andersson wrote: > stdbuf? > unbuffer? none of them is available out of the box on Mac OS X, or FreeBSD. gawk can be installed, but I'd rather use my "while true" loop instead of

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011/9/27 Patrick Proniewski : >> Take a look at a utility called dstat. > no, it's linux only. But it is written in Python - so it should be relatively transportable. I've even managed to modify the code myself - and if I can do it, anybody can! 8-) Paul... > patpro

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

Patrick Proniewski writes: > On 27 sept. 2011, at 20:14, David Garfield wrote: > > > Any entry in a pipe could be buffering. In a quick test here, awk is > > buffering. To find the buffering, try using the pieces up to a given > > stage with " | cat " added at the end. If this buffers, you've

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011/9/27 Patrick Proniewski : > That's what I do, but I think using a loop is ugly, and I would like to find > a way > to feed data continuously into sqlite. I can't see why you would want to do this more than once every minute - or do you? If not, ==

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011/9/27 Patrick Proniewski : > That's what I do, but I think using a loop is ugly, and I would like to find a > way to feed data continuously into sqlite. cron Paul... > patpro -- Hmmm a "life": wonder where I can download one of those? lineh...@tcd.ie Mob:

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On Tue, Sep 27, 2011 at 11:18 AM, Gabor Grothendieck < ggrothendi...@gmail.com> wrote: > On Tue, Sep 27, 2011 at 2:14 PM, David Garfield > wrote: > > Any entry in a pipe could be buffering. In a quick test here, awk is > > buffering. To find the buffering, try

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 09/27/11 20:14, David Garfield wrote: Any entry in a pipe could be buffering. In a quick test here, awk is buffering. To find the buffering, try using the pieces up to a given stage with " | cat " added at the end. If this buffers, you've found the problem. Unbuffered output is usually

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 20:14, David Garfield wrote: > Any entry in a pipe could be buffering. In a quick test here, awk is > buffering. To find the buffering, try using the pieces up to a given > stage with " | cat " added at the end. If this buffers, you've found > the problem. as far as my

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On Tue, Sep 27, 2011 at 2:14 PM, David Garfield wrote: > Any entry in a pipe could be buffering.  In a quick test here, awk is > buffering.  To find the buffering, try using the pieces up to a given > stage with " | cat " added at the end.  If this buffers, you've

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 20:04, Paul Linehan wrote: > 2011/9/27 Patrick Proniewski : > > >> I'm facing a challenging problem. I want to log some data into an SQLite3 DB. >> Data come from a system command (iostat) in an endless steam, one row every >> X seconds: > > > Take a

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

Any entry in a pipe could be buffering. In a quick test here, awk is buffering. To find the buffering, try using the pieces up to a given stage with " | cat " added at the end. If this buffers, you've found the problem. Unbuffered output is usually slower, so it is normally done only to a

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011/9/27 Patrick Proniewski : > I'm facing a challenging problem. I want to log some data into an SQLite3 DB. > Data come from a system command (iostat) in an endless steam, one row every X > seconds: Take a look at a utility called dstat. I've twiddled with the source and

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 18:31, Roger Andersson wrote: > I do not know if tee makes any difference or if it's available on Mac? > http://unixhelp.ed.ac.uk/CGI/man-cgi?tee tee is available, but no more luck here, as it won't allow to disable the buffer. > iostat -d -w 10 disk0 | tee -a logfile >

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 09/27/11 07:48, Patrick Proniewski wrote: I though I could easily pipe data into SQLite: iostat -d -w 10 disk0 |\ awk '!/[a-zA-Z]/ {print "INSERT INTO io VALUES(datetime(\"now\",\"localtime\"),"$1","$2","$3");"}' |\ sqlite3 iostat.db but it won't work, because sqlite3 won't record any

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

g] on behalf of Patrick Proniewski [pat...@patpro.net] Sent: Tuesday, September 27, 2011 6:10 AM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] feed "endless" data into sqlite, thru a shell script On 27 sept. 2011, at 08:31, Baptiste Daroussin wrote: > You don't n

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 Sep 2011, at 1:07pm, Patrick Proniewski wrote: > On 27 sept. 2011, at 13:44, Simon Slavin wrote: > >> If you're using the OS X version, I don't think you need to run iostat as a >> continuous process. Write a shell script with a timed loop which runs >> iostat without the '-w 10'. So

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 13:44, Simon Slavin wrote: > On 27 Sep 2011, at 12:03pm, Patrick Proniewski wrote: > >> You're assuming I'm running Linux, but I'm running Mac OS X Server (or >> FreeBSD by the way), so no /proc here, and iostat is probably working >> differently too. >>

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 Sep 2011, at 12:03pm, Patrick Proniewski wrote: > You're assuming I'm running Linux, but I'm running Mac OS X Server (or > FreeBSD by the way), so no /proc here, and iostat is probably working > differently too. >

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 08:31, Baptiste Daroussin wrote: > You don't need awk :) > > iostat -d -w 10 disk0 | while read a b c; do case $a in *[a-zA-Z]*) > continue ;; *) sqlite3 iostat.db "INSERT INTO io > VALUES(datetime('now', 'localtime'), \"$a\", \"$b\", \"$c\");" ;; > esac; done Ok, this

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 08:02, Stephan Beal wrote: > That's a tricky one, it seems. If you're not restricted to shell code, you > could possibly do this using perl, PHP, or similar. You could open a pipe > for iostat, read a line from the pipe, and feed that line to your db (not in > the form of a

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 12:58, Simon Slavin wrote: > On 27 Sep 2011, at 6:48am, Patrick Proniewski wrote: > >> I've tried various solutions with named pipes, file descriptors >> redirections… but none worked, because they all seem to require the data >> steam to end before feeding data into the

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 Sep 2011, at 6:48am, Patrick Proniewski wrote: > I've tried various solutions with named pipes, file descriptors redirections… > but none worked, because they all seem to require the data steam to end > before feeding data into the DB. Most of your problems are caused because you're

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On 27 sept. 2011, at 08:21, Roger Binns wrote: > The easiest solution is to just be patient and accept the data will be a > little delayed. that won't work for me, because my SQL command includes a datetime('now'). Any row input that is delayed won't be recorded with the proper datetime. That's

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

2011/9/27 Patrick Proniewski : > Hello, > > I'm facing a challenging problem. I want to log some data into an SQLite3 DB. > Data come from a system command (iostat) in an endless steam, one row every X > seconds: > >         disk0 >   KB/t tps  MB/s >   4.02 2318 9.09 >   4.00

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/26/2011 10:48 PM, Patrick Proniewski wrote: > but it won't work, because sqlite3 won't record any data until the iostat > command ends. UNIX tools using the standard I/O library will show this. They detect that standard output is not a

Re: [sqlite] feed "endless" data into sqlite, thru a shell script

On Tue, Sep 27, 2011 at 7:48 AM, Patrick Proniewski wrote: > while true; do > ... endless loop, forking iostat for 2 rows of data (-c 2), keeping only the > last row because the first one is an artifact (tail -1). > That's a tricky one, it seems. If you're not restricted to

[sqlite] feed "endless" data into sqlite, thru a shell script

Hello, I'm facing a challenging problem. I want to log some data into an SQLite3 DB. Data come from a system command (iostat) in an endless steam, one row every X seconds: disk0 KB/t tps MB/s 4.02 2318 9.09 4.00 1237 4.83 6.63 979 6.34 46.30 15 0.69 30.58