On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Yes, I do wish to inverse the order, but the forward in time file
will be in binary.
I really think it will be simplest to just write the file in forward
order, then use mmap to read it one record at a time. It might
Data files have some sort of parsing, unless it's one huge dict, or
list, so there has to be an average size to the parse.
Not meaning the dict, or list isn't parsed, but that the file should
be have parsable areas.
--
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
--
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Yes, I do wish to inverse the order, but the forward in time file
will be in binary.
I really think it will be simplest to just write the file in forward
order, then use
On 28-Oct-2012 12:18, Dave Angel wrote:
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Yes, I do wish to inverse the order, but the forward in time file
will be in binary.
I really think it will be simplest to just
On 28 October 2012 14:20, Virgil Stokes v...@it.uu.se wrote:
On 28-Oct-2012 12:18, Dave Angel wrote:
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Yes, I do wish to inverse the order, but the forward in time file
On 2012-10-28 19:21, Oscar Benjamin wrote:
On 28 October 2012 14:20, Virgil Stokes v...@it.uu.se wrote:
On 28-Oct-2012 12:18, Dave Angel wrote:
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Yes, I do wish to inverse
On 24-Oct-2012 17:11, rusi wrote:
On Oct 23, 7:52 pm, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data
On 24-Oct-2012 00:36, David Hutto wrote:
Don't forget to use timeit for an average OS utilization.
I'd suggest two list comprehensions for now, until I've reviewed it some more:
forward = [%i = %s % (i,chr(i)) for i in range(33,126)]
backward = [%i = %s % (i,chr(i)) for i in range(126,32,-1)]
On 24-Oct-2012 02:06, Oscar Benjamin wrote:
On 23 October 2012 15:31, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB) that contain time
series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
format. I perform various types of processing
On 24-Oct-2012 00:57, Demian Brecht wrote:
This is a classic example of why the old external processing algorithms
of the 1960s and 70s will never be obsolete. No matter how much memory
you have, there will always be times when you want to process more data
than you can fit into memory.
But
On 24-Oct-2012 00:53, Steven D'Aprano wrote:
On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB)
[...]
Finally, to my question --- What is a fast way to write these
On Wed, Oct 24, 2012 at 3:05 AM, Virgil Stokes v...@it.uu.se wrote:
On 24-Oct-2012 00:36, David Hutto wrote:
Don't forget to use timeit for an average OS utilization.
I'd suggest two list comprehensions for now, until I've reviewed it some
more:
forward = [%i = %s % (i,chr(i)) for i in
On Wed, Oct 24, 2012 at 3:17 AM, Virgil Stokes v...@it.uu.se wrote:
On 24-Oct-2012 00:57, Demian Brecht wrote:
This is a classic example of why the old external processing algorithms
of the 1960s and 70s will never be obsolete. No matter how much memory
you have, there will always be times
On 23-Oct-2012 22:03, Cousin Stanley wrote:
Virgil Stokes wrote:
Not sure about tac --- could you provide more details on this
and/or a simple example of how it could be used for fast reversed
reading of a data file ?
tac is available as a command under linux
$ whatis tac
tac
On Wed, 24 Oct 2012 01:23:58 -0400, Dennis Lee Bieber wrote:
On Tue, 23 Oct 2012 16:35:40 -0700, emile em...@fenx.com declaimed the
following in gmane.comp.python.general:
On 10/23/2012 04:19 PM, David Hutto wrote:
forward = [line.rstrip('\n') for line in f.readlines()]
f.readlines()
On 24/10/2012 08:07, Virgil Stokes wrote:
On 23-Oct-2012 22:03, Cousin Stanley wrote:
Virgil Stokes wrote:
Not sure about tac --- could you provide more details on this
and/or a simple example of how it could be used for fast reversed
reading of a data file ?
tac is available as a
On 10/23/2012 4:35 PM, emile wrote:
So, let's see, at that point in time (building backward) you've got
probably somewhere close to 400-500Gb in memory.
My guess -- probably not so fast. Thrashing is sure to be a factor on
all but machines I'll never have a chance to work on.
I went
On 2012-10-23, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote:
I would be very surprised if the poster will be able to fit 100
gigabytes of data into even a single list comprehension, let alone
two.
This is a classic example of why the old external processing
algorithms of the
Emile van Sebille em...@fenx.com writes:
probably somewhere close to 400-500Gb in memory
I went looking for a machine capable of this and got about halfway
there with http://www.tech-news.com/publib/pl2818.html which allows up
to 248Gb memory -- near as I can tell the price for the maxed
On Oct 23, 7:52 pm, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB) that contain time
series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I
perform
various types of processing on these data (e.g. moving median, moving
I am working with some rather large data files (100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data (e.g. moving median, moving average,
and Kalman-filter, Kalman-smoother) in a sequential
On 10/23/12 09:31, Virgil Stokes wrote:
I am working with some rather large data files (100GB) that contain time
series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I
perform
various types of processing on these data (e.g. moving median, moving
average,
and
Virgil Stokes v...@it.uu.se writes:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in backwards?
Seeking backwards in files works, but the performance hit is
significant. There is also a performance hit to scanning pointers
Paul Rubin no.email@nospam.invalid writes:
Seeking backwards in files works, but the performance hit is
significant. There is also a performance hit to scanning pointers
backwards in memory, due to cache misprediction. If it's something
you're just running a few times, seeking backwards the
On 10/23/12 11:17, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in backwards?
Seeking backwards in files works, but the performance hit is
significant. There is also a
Tim Chase python.l...@tim.thechases.com writes:
Again, the conversion to/from decimal hasn't been a great cost in my
experience, as it's overwhelmed by the I/O cost of shoveling the
data to/from disk.
I've found that cpu costs both for processing and conversion are
significant. Also, using a
On 23-Oct-2012 18:09, Tim Chase wrote:
On 10/23/12 09:31, Virgil Stokes wrote:
I am working with some rather large data files (100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data (e.g. moving
On 23-Oct-2012 18:17, Paul Rubin wrote:
Virgil Stokes v...@it.uu.se writes:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in backwards?
Seeking backwards in files works, but the performance hit is
significant. There is also a
On 23-Oct-2012 18:35, Dennis Lee Bieber wrote:
On Tue, 23 Oct 2012 16:31:17 +0200, Virgil Stokes v...@it.uu.se
declaimed the following in gmane.comp.python.general:
Finally, to my question --- What is a fast way to write these variables to an
external file and then read them in backwards?
On 10/23/12 12:17, Virgil Stokes wrote:
On 23-Oct-2012 18:09, Tim Chase wrote:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in
backwards?
Am I missing something, or would the fairly-standard tac
utility do the reversal you
On 23-Oct-2012 19:56, Tim Chase wrote:
On 10/23/12 12:17, Virgil Stokes wrote:
On 23-Oct-2012 18:09, Tim Chase wrote:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in
backwards?
Am I missing something, or would the
Virgil Stokes wrote:
Not sure about tac --- could you provide more details on this
and/or a simple example of how it could be used for fast reversed
reading of a data file ?
tac is available as a command under linux
$ whatis tac
tac (1) - concatenate and print files in reverse
On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB) that contain time
series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
format. I perform various types of processing on these data (e.g. moving
median,
Don't forget to use timeit for an average OS utilization.
I'd suggest two list comprehensions for now, until I've reviewed it some more:
forward = [%i = %s % (i,chr(i)) for i in range(33,126)]
backward = [%i = %s % (i,chr(i)) for i in range(126,32,-1)]
for var in forward:
print
Missed the part about it being a file. Use:
forward = [%i = %s % (i,chr(i)) for i in range(33,126)]
backward = [%i = %s % (i,chr(i)) for i in range(126,32,-1)]
print forward,backward
This was a dud, let me rework it real quick, I deleted what i had, and
accidentally wrote the wrong
On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB)
[...]
Finally, to my question --- What is a fast way to write these variables
to an external file and then read
This is a classic example of why the old external processing algorithms
of the 1960s and 70s will never be obsolete. No matter how much memory
you have, there will always be times when you want to process more data
than you can fit into memory.
But surely nobody will *ever* need more than
Whether this is fast enough, or not, I don't know:
filename = data_file.txt
f = open(filename, 'r')
forward = [line.rstrip('\n') for line in f.readlines()]
backward = [line.rstrip('\n') for line in reversed(forward)]
f.close()
print forward, \n\n, \n\n, backward, \n
--
On Tue, Oct 23, 2012 at 6:53 PM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB)
[...]
Finally, to my
On 10/23/2012 04:19 PM, David Hutto wrote:
Whether this is fast enough, or not, I don't know:
well, the OP's original post started with
I am working with some rather large data files (100GB)...
filename = data_file.txt
f = open(filename, 'r')
forward = [line.rstrip('\n') for line in
Virgil Stokes v...@it.uu.se writes:
Yes, I do wish to inverse the order, but the forward in time file
will be in binary.
I really think it will be simplest to just write the file in forward
order, then use mmap to read it one record at a time. It might be
possible to squeeze out a little more
On Tue, Oct 23, 2012 at 7:35 PM, emile em...@fenx.com wrote:
On 10/23/2012 04:19 PM, David Hutto wrote:
Whether this is fast enough, or not, I don't know:
well, the OP's original post started with
I am working with some rather large data files (100GB)...
Well, is this a dedicated system,
On 23 October 2012 15:31, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB) that contain time
series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
format. I perform various types of processing on these data (e.g. moving
median, moving
On 10/23/12 13:37, Virgil Stokes wrote:
Yes, I do wish to inverse the order, but the forward in time
file will be in binary.
Your original post said:
The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format
making it hard to know what sort of data is in this file.
So I guess it
On Tue, Oct 23, 2012 at 8:06 PM, Oscar Benjamin
oscar.j.benja...@gmail.com wrote:
On 23 October 2012 15:31, Virgil Stokes v...@it.uu.se wrote:
I am working with some rather large data files (100GB) that contain time
series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
format.
45 matches
Mail list logo
