Tomasz,
Thank you for taking the time to understand what I was getting at.
You are right that such a function should only be used by those who
fully understand their responsibilities to update the array at the
appropriate times, and can debug their own scripts with that in
mind. Making it hidden --only for those who need it to do
sophisticated real time AFLs is one solution. This would solve half
my problems. Perhaps a "fast array valid" flag in AFL could ease
some of the potential errors by the user for common conditions.
Do you have any thoughts about the feasibility of solutions for the
second problem set?
Best regards,
Dennis Brown
On Aug 5, 2007, at 5:17 AM, Tomasz Janeczko wrote:
Hello,
Regarding ATC bar requirements: that's true because ATC is designed
to create composites and
for consistency it always requests all bars so composite is
complete (generated from all quotes, not just from "viewing" area).
This behaviour however can be turned off using SetBarsRequired()
placed at the *end* of the formula
(that will overwrite any AB's own calculations). See http://
finance.groups.yahoo.com/group/amibroker/message/113635
for more details.
From what I read below, you want "static array" to be used ONLY
when BarCount does not change and symbol does not change.
Well - this could work without synchronization and all
"bookkeeping" I mentioned below and I guess it should solve all
your problems
because it would be roughly as fast as other arrays, BUT....I have
only one problem with this idea - that I will get endless
"bug reports" from people not reading the docs even if I write in
big red letters that static arrays do not perform any checking/
aligninig/ padding
the way foreign() works and that the user is responsible for using
it wisely. I am not speaking about you or anyone else experienced
in AB, because I am sure you know
what you are doing, but I mean newbies :-)
Maybe make it "hidden" functionality???
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: Dennis Brown
To: [email protected]
Sent: Sunday, August 05, 2007 3:53 AM
Subject: Re: [amibroker] Second thoughts on Static arrays --whole
problem set
Tomasz,
First let me thank you for your patients in explaining this one
more time. It makes a very good post for the benefit of others.
Yes, I am aware of everything you explained because you had
explained it before and I listened and tried out the ideas.
I sent you a direct email through support #39911 on May 16 for
which I received no reply (and none was technically required). I
will quote it:
"I am taking this subject off the yahoo board now because I don't
think anyone can help but you.
It can go back on the board when there is a solution that will work.
I took your advice and made all the changes you suggested to speed
up my AFL.
The functionality of the AFL is working nicely, behaving just as it
should.
However, instead of running 10 times faster, now it runs twice as
slow as before.
I have narrowed the problem down to the very thing that was
supposed to speed it up.
1. When I use the ATC & Foreign functions the number of bars
required goes from 3518, 0 to 1000000, 1000000.
This looks like it turns off fast AFL and slows all normal
processing by requiring all the bars. I have 77,000 data bars total.
I do not know if this is intentional or a bug.
2. The ATC function is slow, but I only do it once per bar (1
minute).
3. The foreign function which is done every time is slow also
(relative to normal arrays).
It takes .04 seconds each and I need dozens to replace the missing
static array functionality.
Perhaps this is also due to the 1000000 bars.
Without being able to have persistent arrays that are as fast as
normal arrays I will not be able to achieve my trading goals for
very many years until computers speed up by 10x. That is the same
as never to me. I wish I could just declare some variables to be
Persistent or Static and be done with it. It is easy for me to
say, but perhaps it is not so easy for you to do. However, being
able to successfully use the methods discussed on the board opens
up a lot of extra possibilities due to the extra speed possible
with AFL.
I appreciate any help you can give in this matter."
As I want a workable solution to my problem and want to use AB for
my project (I would be trading this already using my AB system if I
had a solution), I will state the fundamental problems I face
clearly together here (instead of disjointed posts like before) and
then perhaps a solution might occur to you or somebody else.
Please forgive the length and complexity of this post.
Background:
My trading system is very simple in principle. AFL calculates a
complex indicator, I draw manual trend lines, and I place manual
trades on a single stock or ETF at a time. It is day trading, so I
only have a few seconds to trigger my trade when the conditions are
right. My AFL allows rapid manual selection of different
timeframes via a parameter, and selects the proper volume bar
settings dynamically for different ETFs or stocks. The program
automatically runs through many timeframes to select trade trigger
possibilities for me to consider.
1. There is a heavy compute environment --several seconds of AFL.
2. The environment is highly interactive --drawing trend lines.
3. Seconds count for placing trades.
First Problem set:
Under heavy compute loads (when AFL time is greater than refresh
rate), AB stops responding to the UI.
Workarounds:
The only way I can get some UI functionality back is to set the
refresh rate to one second longer than the AFL compute time. Then
I can get UI functionality back, but with many delays.
I also have a ParamToggle() to turn off most indicator processing
so that I can interact with the UI better at times (but I lose my
indicator plot references).
This is a partially effective workaround that I could limp along with.
Requested solution:
Realizing that the root problem is that the UI functions only run
(or initiate) in between AFL passes, I looked for a way to do less
processing on each pass. The AFL load could be much less most of
the time because I only need to calculate everything when a new bar
is added.
I thought of two possibilities:
1. Ask for an AFL function that is essentially DoUserInterface().
Then put AFL into an infinite loop and put these commands at
appropriate places. Since this is backwards from the way the AFL
appears to work, I did not make this suggestion.
2. Ask for arrays that keep their data between AFL passes. I
tried using ATC for this. It functioned like it was supposed to,
but was too slow to be a practical solution. I understand the
reason they are so slow, and that is because they are a true static
permanent array. However, what I wanted was only for a normal
array to be saved between passes. They only need to be identified
by which pane they are associated with (could be part of the
internal name). They are only valid between new bars (same array
size). Changes to symbol, timeframe, parameters, etc., can be
detected in the AFL and the array updated. I know this can be
done, because I did it for the ATC version I wrote and everything
worked perfectly (just not faster). Because AFL is so fast working
with arrays, It seems that it should be possible to just save an
array in memory in a high speed way, then recall it to repopulate
the array instead of recalculate the array if the AFL determines
that the data would not have changed since the last pass.
There may be another possible solution that I have not thought of
and I would love to hear it.
Second Problem set:
Because this is a realtime trading system, decisions have to be
made from raw data which contains bad ticks that may fool the system.
Volume bars are used in the system, and must be able to change
volume per bar on different AFL passes.
Built-in timeframe or volume settings do not allow the flexibility
to address these
Workarounds:
A tick database is too large to have enough history for the
indicators. Running a database of 5 seconds for a basic 1 minute
bar (or equivalent for volume bars) allows for reasonably effective
bad tick removal and volume bar generation. History can be just
60K bars. Bad ticks are removed and volume bar settings are
determined using the 5 second bars. Arrays are built up using
TimeFrameSet() to get the actual bars used for indicator
calculations and graph plots. This approach works really well,
however, there is one big problem with it.
Even though I can display all my price and indicators in the
timeframe I desire, I can not draw studies that align to that
timeframe. All studies are drawn in the pane relative to 5 second
bars. This workaround is not acceptable for trading.
Requested solution:
#1117 Make an AFL way to set the timeframe of the drawing studies
to match the OHLC array created by TimeFrameSet().
Of course implementing this has a general benefit of giving an
almost infinite number of time/volume/range bars settings via AFL
and parameters.
Please consider the problems (not just proposed solutions) and
advise if other possibilities come to mind for solutions.
Thank you very much for your patients and consideration,
Dennis Brown
On Aug 4, 2007, at 6:16 AM, Tomasz Janeczko wrote:
Dennis,
I explained it many times:
1. You can not simply "KEEP" the array from previous execution and
reference it directy becase:
a) previous execution may be on different symbol/different time
frame/different zoom level leading
to DIFFERENT NUMBER OF ELEMENTS in the array and different
TIMESTAMPS of array bars.
2. Because of (1) the only way to implement "static array" is to:
a) store not only array BUT also TIMESTAMPS with every bar
b) during reading perform timestamp checking and synchronization
so referenced array data is aligned
with current execution data
If those two were not implemented you would NOT be able to use
"static arrays" at all.
3. Functionality already exists
a) Point 2a - storing arrays with timestamps is exactly what
AddToComposite ALREADY DOES
b) Point 2b - reading arrays with timestamps and aligning is
exactly what Foreign ALREADY DOES
4. Implementing dedicated "static" array functions would
a) give NO SPEED ADVANTAGE over AddToComposite/Foreign
b) give only "the impression" that it is something else and
"faster" than ATC/Foreign while it would
be FALSE impression.
Summary:
a) functionality already exists
b) no speed advantage over existing functionality
So here are the functions you are asking for:
function StaticVarArraySet( Varname, array )
{
AddToComposite( array, "~SA_"+VarName, "C", atcFlagDefaults |
atcFlagEnableInBacktest | atcFlagEnableInExplore |
atcFlagEnableInIndicator | atcFlagEnableInPortfolio );
}
function StaticVarArrayGet( VarName );
{
return Foreign( "~SA_"+VarName, "C" );
}
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: Dennis Brown
To: [email protected]
Sent: Saturday, August 04, 2007 5:26 AM
Subject: [amibroker] Second thoughts on Static arrays
Hi,
The more I have thought about static arrays for my AFL needs, the
more I see that what I am looking for is a quite limited
implementation.
Static arrays would have low overhead and give the speed needed
just like regular arrays. The primary need is to avoid re-
computing complex things all the time that only need to be
computed under certain conditions or perhaps once per new bar.
This allows far richer AFLs to be developed for real time trading,
while improving the responsiveness of the UI interaction under
heavy compute conditions. The life of a static array is fleeting.
Static arrays are not permanent like ATC, and do not have the big
overhead. Static arrays as I envision them, would have to be
updated via the user AFL program whenever a change in environment
caused the data to become invalid --like changing the symbol or
the timeframe --or even adding a new bar (which changes the array
size). Having static arrays like this would allow breaking the
AFL into two parts --the fast part that needs to run for every
second, and the part that needs to run once per compressed
timeframe, volume, or range bar.
I am sure there are technical issues involved, or TJ would have
already provided them since they are so useful for speeding up
AFL. First rule of making programs fast is pre-compute everything
possible and not inside a loop. AFL is one big loop.
Perhaps what I am looking for are not technically "static" arrays
in the normal sense. Maybe I need to call them by a different
name for the suggestion box. Perhaps there is an even easier way
to address this need with a new AFL syntax: keep(myArray);
Comments anyone?
Dennis Brown
