On 28 Apr 2005, at 17:59, Dennis Brown wrote:
Frank,
This is a simplified algorithm that I think would best translate
directly between Transcript and PostgreSQL for a time trial. Just
make sure the PostgreSQL code is an item by item algorithm also.
Thanks,
Dennis
Transcript output on my
On 28 Apr 2005, at 17:59, Dennis Brown wrote:
Frank,
This is a simplified algorithm that I think would best translate
directly between Transcript and PostgreSQL for a time trial. Just
make sure the PostgreSQL code is an item by item algorithm also.
Thanks,
Dennis
Transcript output on my
Read the earlier versions of this thread for the fast methods. This
example is just for a specific test between Transcript and PostgreSQL
Dennis
On Apr 29, 2005, at 6:24 AM, Wouter wrote:
On 28 Apr 2005, at 17:59, Dennis Brown wrote:
Frank,
This is a simplified algorithm that I think would
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Did my earlier reply even make it through?
I'm not seeing it on the list. Transcript is actually faster than what
I was able to come up with for PostgreSQL -- much faster, in fact.
Ada is WAY faster than either one of them -- a fraction of a second
Subject: Re: Making Revolution faster with really big arrays
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Did my earlier reply even make it through?
I'm not seeing it on the list. Transcript is actually faster than what
I was able to come up with for PostgreSQL -- much faster, in fact.
Ada is WAY
Frank,
Your earlier reply did not make it through.
This is kinda what I thought would be the case, even though I was
hoping for better. It seems that the way to go is to use another
compiled language for the number crunching and Rev for the UI. This is
doable in my case because I write out
Jon,
I agree, this would be worthwhile to come up with. In my case all I
really need is a minimal compiled language that can read a disk file,
do a bunch of array math, and write the results back out. Don't really
need the usual complexities of the GUI etc. Any suggestions?
Dennis
On Apr
Jon,
It should actually be very easy to create such a thing. This is
something I have requested as far back as 6 or 7 years ago for
Transcript (I was using SC back then). I was told that Keyed Arrays
were coming and would solve all my problems... think again. It is no
different than having
Frank,
That sounds interesting. How fast can PostgreSQL generate the simple
example I showed using an item by item algorithm? Will it really
execute faster than Transcript?
Dennis
On Apr 27, 2005, at 6:39 PM, Frank D. Engel, Jr. wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Try
faster with really big arrays
Frank,
That sounds interesting. How fast can PostgreSQL generate the simple
example I showed using an item by item algorithm? Will it really
execute faster than Transcript?
Dennis
On Apr 27, 2005, at 6:39 PM, Frank D. Engel, Jr. wrote:
-BEGIN PGP SIGNED MESSAGE
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I haven't actually tried it with this type of algorithm, so I really
don't know, and I've already deleted the eMail with the sample code;
you'd need to send that code out again for me to even try testing it
for you.
The docs for pl/pgsql are part
On Apr 28, 2005, at 10:18 AM, Frank D. Engel, Jr. wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I haven't actually tried it with this type of algorithm, so I really
don't know, and I've already deleted the eMail with the sample code;
you'd need to send that code out again for me to even
Hi Dennis,
Here's an approach that uses keyed arrays. Run time on an old iMac
with 320 MB RAM and a 400 Mhz clock is 7.2 seconds.
on mouseUp
global gTestArrayX,gTestArrayY
if gTestArray is empty then
repeat with i = 1 to 1000
repeat with j = 1 to 1000
put j into
Sorry to disappoint Rob,
Make your script actually do some work and it runs a lot slower. I
added the following two handlers to my menu button and the results
were:
23 seconds to init the array and 20 sec to process (on a 1.8GHz G5). I
have to add the array initialization time to your example
Dennis,et al:
I don't think you can beat the repeat for each, but I would certainly like
to see it if you could.
I don't think repeat for each will be beaten, either. I wws mainly
interested in how keyed access stacked up against the other options you tried.
Rob Cozens CCW
Serendipity Software
Stephen,
The math I want to do is not regular --as in the simple example, but
very algorithmic on a item by item basis. The overhead of passing the
data to and from MySQL will kill the speed unless it can do all the
algorithms internally. Previous attempts have shown this.
Dennis
On Apr 26,
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Try PostgreSQL, which offers stored procedures. You could write the
complex parts of the algorithms in pl/pgsql, if nothing else -- or
pl/perl, or pl/python, or whatever.
On Apr 27, 2005, at 12:07 PM, Dennis Brown wrote:
Stephen,
The math I want to
On Apr 14, 2005, at 10:09 AM, Dennis Brown wrote:
Dick,
Thanks for your various tips and help online and offline. I need to
take a week to digest and incorporate the ideas into a real piece of
code that I hope will fly like the wind --then I will share my code
and a data generator for you and
MySQL can do math on columns like you want. Very Fast. No loops in transcript.
Check out the GROUP BY (Aggregate) Functions.
http://dev.mysql.com/doc/mysql/en/group-by-functions.html
At 8:06 PM -0400 4/26/05, Dennis Brown wrote:
Can anyone think of another way to do this that would be faster
Dick,
Thanks for your various tips and help online and offline. I need to
take a week to digest and incorporate the ideas into a real piece of
code that I hope will fly like the wind --then I will share my code and
a data generator for you and everyone else to see and rip to shreds ;-)
Thanks
On 4/12/05 7:36 PM, Dennis Brown [EMAIL PROTECTED] wrote:
...I have two arrays 10,000 lines with 2500 items in each line...
...I need to pair the items together...
Do you have something like closing prices in one container and shares traded
in another container, each for the same 10,000
Dennis Brown wrote:
Thanks Alex,
This is quite an imaginative suggestion. It would work great if I
just had two lists, but I have two arrays 10,000 lines with 2500 items
in each line. It would be a lot more complex to combine them into a
single array (I need to pair the items together). But
To Alex Dick,
Yes, I am doing something like what Dick suggested, only a lot more
complex --actually I have about a hundred very different calculations
to do on various arrays which is why I was not more specific about the
calculations (it has to do with generating various statistics and
Dennis,
I guess it's half of each, as I see it (and I misread some of this I
think).
You only need sequential access to the lines and items, but random
access would solve your problems. Random access is even faster than
sequential, and can do everything sequential can.
From your syntax
Brian,
Can you create a 1 by 2500 by 100 array of random numbers (sparsely
populated i.e. some values are null) in SQL?
Can you put the sum of two random accessed numbers (except if one is
null then the sum is null) into a third random location?
How fast can you do this last thing 2500
On Tuesday, April 12, 2005, at 07:25 PM,
[EMAIL PROTECTED] wrote:
I might be able to suffer with the
chunk specification for the line#, then use a repeat for each item and
put 2500 items in an array. That way I will only need 2500 array items
at any one time instead of 125,000,000 array items
Mark,
Yes, this is what I am going to do.
Thanks,
Dennis
On Apr 13, 2005, at 1:37 PM, Mark Brownell wrote:
On Tuesday, April 12, 2005, at 07:25 PM,
[EMAIL PROTECTED] wrote:
I might be able to suffer with the
chunk specification for the line#, then use a repeat for each item and
put 2500 items in
On Wednesday, April 13, 2005, at 10:37 AM, Mark Brownell wrote:
Dennis,
I once used the split function to create almost instant arrays based
on 1, 2, 3, etc... as the locations created by the split, ( I call
them locations because I'm an old Director user.) This process could
work well when you
Dennis,
I through together a real rough test- just some tables with random
values 0-1000.
SELECT (v1 + v2500) FROM values1,values2500 LIMIT 50;
This ran for me in about 10-15 seconds off of a G4 XServe with 1GB
memory. Note that I only had enough free memory to do chunks of 500,000
Brian,
I am running on a 1.8 GHz single processor G5 Tower.
With my data put into an array it takes about 9 sec to access each
value.
With my data in a variable, it takes about 1.5 sec to access each value
sequentially.
I don't have the new approach written yet (got side tracked on the
graphing
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Rev's arrays are associative. When using an array with an index like
[X, Y, Z], you are really saying, make a string whose contents are X,
Y, and Z separated by commas, then use that as the index for the array.
These array indexes take up memory,
Welcome to the Revolution Dennis,
Why could you not take help from the native Rev ablity to manage the
process in storing the datas inside an ACID-DB alike PostgreSQL or
OpenBase ? It's how i would handle such amounts of datas, for my own.
Transcript for the RAM fine high-speed calculations and
Thanks Frank,
Actually I have a 3.5GB 64 bit G5 machine that can handle that much
data, and I could add a couple more gig if I needed to. It crashes
when I get less than 1GB into RAM (I can monitor the number of free
pages of RAM). I tried processing it like you suggest. However, at
the
Thanks Pierre,
I considered that also. A Database application would certainly handle
the amount of data, but they are really meant for finding and sorting
various fields, not for doing the kind of processing I am doing. The
disk accessing would slow down the process.
Dennis
On Apr 12, 2005,
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
That is only significant if Rev takes advantage of the 64-bit address
space, which I seriously doubt. Your Rev process will still be limited
to 2GB of addressing space, regardless of how much RAM is in the
system. Until they release a 64-bit
Frank D. Engel, Jr. wrote:
That is only significant if Rev takes advantage of the 64-bit address
space, which I seriously doubt. Your Rev process will still be limited
to 2GB of addressing space, regardless of how much RAM is in the
system.
Wouldn't that be 4GB, or have I slipped a digit?
--
I would not disregard this option so readily. You can use the
database as an array of sorts allowing you to pull out any set of
data you need to process and store the results there for later. If
you are concerned with performance of server-based databases like
PostgreSql, OpenBase, or MySql,
Hi Dennis,
I have found that large data files can be broken down into smaller
objects using simplified XML where access is obtained using a
pull-parser. Unlike the XML parser in Revolution a very fast
pull-parser can be used to break down objects and parse out specific
fields without ever
I would not disregard this option so readily. You can use the
database as an array of sorts allowing you to pull out any set of
data you need to process and store the results there for later. If
you are concerned with performance of server-based databases like
PostgreSql, OpenBase, or MySql,
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
A 32-bit address space has a *total* of 4GB of addressable memory. Of
that 4GB, the operating system normally reserves 2GB for its own use,
leaving only 2GB for the process. The operating system reserves that
much space because it maps the SAME
On Apr 12, 2005, at 7:37 PM, Dennis Brown wrote:
Frank,
I just copied this from the Rev IDE help info:
A note about entries designated as Unlimited:
Since each open stack file resides completely in memory, Revolution
stacks (and all structures within a stack) are effectively limited by
available
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Rev is still confined to what the Operating System offers it, which is
2GB for most 32-bit systems, and never 4GB (3GB, or maybe even 3.5GB in
some rare cases), as per my previous note.
I just noticed that Tiger hits the shelves in 17 days. Sweet.
Great info on the memory limits, Frank. Thanks for that.
Frank D. Engel, Jr. wrote:
Again, though, with processor-intensive operations on huge data sets,
*any* scripting language is a poor choice. You definitely want
native-compiled code for that. Pascal, Ada, a compiled BASIC...
Of course
Dennis Brown wrote:
Hi all,
I just joined this list. What a great resource for sharing ideas and
getting help.
So I came up with an idea for a proposed language extension. I
put the idea in Bugzilla yesterday, then today, I thought I should ask
others if they liked the idea, had a
Thanks Mark,
Your suggestion would not help for my application, the offset function
is not faster than counting up returns (line chunk statement) to get to
the proper line, but I have used it before for parsing a large USDA
food nutrition database and it helped a lot. However, it does give me
Dennis,
I have to agree with Pierre here. If you are looking for random access
to many thousands of records taking up gigabytes of memory, a database
engine is, IMO, the only logical choice.
A simple MySQL/PostgreSQL/Valentina/etc database indexed by line number
(or stock symbol) would be very
Thanks Brian,
I don't require random access to the data. I only need sequential
access. That is why the repeat for each operator works so fast --less
than a microsecond per data item. I'm not going to match that with
anything other than RAM.
Dennis
On Apr 12, 2005, at 10:06 PM, Brian Yennie
Thanks Alex,
This is quite an imaginative suggestion. It would work great if I just
had two lists, but I have two arrays 10,000 lines with 2500 items in
each line. It would be a lot more complex to combine them into a
single array (I need to pair the items together). But it could be done
Dennis, this may not be directly relevant, but I've made a backtesting
app for futures which works reasonably quickly. It uses daily data from
50 or so markets going back, in some cases, to the mid 70's. If you're
dealing with intraday data and thousands of stocks then obviously
that's many,
49 matches
Mail list logo
