Re: [Freedos-devel] Extension proposal

[Bret Johnson]

> Sometimes, I suffer from a severe lack of trust in other
> programmers. I look at the "Install Check" and see only specific
> registers are to be modified and no others are to be destroyed. But
> in my head, I think that someone out there may think it is a good
> idea to save some time by passing some additional information back
> in one of those registers. However, that would be their bug and not
> mine. So, I will remove them and not worry about it (very much). :-)

That was one of my problems when I was using INT 2Fh since it doesn't really 
have a defined technical specification.  Sometimes, even with MS programs, even 
the "Install Check" function will return data in various registers (like a 
version number or something).  When I was using INT 2Fh I needed to be very 
cautious since I never knew what unexpected things some other program might do. 
 AMIS (and IISP) are rigidly defined so there shouldn't be any "surprises".  
AMIS also lets you define your own sub-functions and those can work however you 
want them to, but the defined functions need to follow the specification 
precisely.

In addition, I should also warn you that if you plan on using AMIS in the 
future you should probably start doing it right away instead of waiting.  This 
is a problem I had with some of the programs I've had out for a long time 
(since before I started using AMIS).  In the old versions of the programs I 
used INT 2Fh but the newer versions use AMIS (INT 2Dh).  The problem is that 
when I'm trying to install a new version of the program I need to test to see 
if another copy of the program is already installed, and if so which version it 
is.  To make sure a really old version isn't already installed I need to do 
both the old (INT 2Fh) and new (AMIS/INT 2Dh) tests.  That is both cumbersome 
and slow and should be avoided.  You usually don't want two versions of the 
same program running at the same time unless they are specifically designed to 
"cooperate" with each other.


___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[jerome]

Hi ecm, 

> On Apr 24, 2023, at 12:57 PM, C. Masloch  wrote:
> [..]
> 
>> 2) We could walk the device driver chain. Thats fairly straight forward and 
>> easy enough to implement. Lets do that… Hmmm, not all device drivers are 
>> showing up in the chain and logger is not being seen. Let’s not worry about 
>> that for now and do something else.
> 
> Don't know how you managed to do that, it is indeed straightforward.

I should have kept the original one that failed to work. So, I could look back 
and laugh at how I managed to mess it up. However, it could have been related 
to doing some of the testing inside DOSBox. Oh well, doesn’t matter now. 

>> [..]
>> I did consider walking the MCB chain to find it. But, that comes with its 
>> own set of problems. Some blocks contain sub-blocks.
> 
> SD "system data" blocks with sub-MCBs are a fact on most kernels, but they're 
> not terribly complicated to detect and handle. First two name bytes are "SD" 
> and the owner is 8. Sub-MCBs have signature letters indicating the type of 
> the block rather than "M" link / "Z" no link, and they always span the entire 
> space up to the end of the container SD block.
> 
>> The upper memory blocks may or may not be linked into the primary MCB chain.
> 
> This is more of a problem. You can ignore a "Z" (ie, treat it as if it is an 
> "M") if you know that the next block after the "Z" block is the first UMCB. 
> You can find the first UMCB by trying to call interrupt 2Fh, function 1261h, 
> CY; if it returns NC then ax = first UMCB. Or if the kernel does not support 
> that function, switch off the UMB link, walk until "Z", switch on the UMB 
> link, walk until you're past the MCB that was "Z" prior to this (but is now 
> "M"). Of course, you want to preserve the original UMB link state around 
> this. Example code is in my TSR example [1].
> 
> For a single search, it may be more efficient to just do the UMB link state 
> enabling around your MCB walker as is, and honor any "Z" as the final end 
> marker then.
> 
> 
> [1]: https://hg.pushbx.org/ecm/tsr/file/749e0f25364c/transien.asm#l96

Yup. Decades ago I wrote an MCB walker in Turbo Pascal. If it can be done in an 
HLL, it can be done in assembly. :-)

>> [..]
>> For “fun”, I implemented initial versions of both the 0x2b interrupt hook 
>> and a partial 0x2d implementation to locate the Logger device driver. The 
>> 0x2d version is very bare bones and does not include several functions 
>> required to be “fully compliant”. It only responds to the install check 
>> function for whichever multiplexer it allocates. Other function requests 
>> only respond with AL=0x00 which i guess is “not implemented”.
> 
> Yes, setting al to zero is the way to indicate a function is not implemented.

Looking at RBIL, it says that it returned al=zero for the “reserved” functions.

So, I assumed that would be the case for standard functions (0x01-0x06) that 
are not supported. 

Thanks for the clarification.

> 
>> As such, 0x2d uses 35 bytes more resident memory than 0x2b. (actually 49 
>> bytes more if you count a “title” string that is not required). If I spend 
>> the time making it fully compliant, it will require even more. It really 
>> would need function 04 (determine chained ints) and should have 06 (device 
>> driver info).
> 
> Why not show your examples? Perhaps we can point out some optimisations to 
> shave off a few bytes here and there. I have some ideas about how to write a 
> very small int 2Dh handler, eg:
> 
> int_2D:
>  ; IISP header here
> 
>  ; magic byte sequence common to Ralf Brown and ecm programs:
>  cmp ah, 0
> .multiplex_number: equ $ - 1
>  ; SMC, storing the multiplex number in the instruction immediate
>  je .handle
>  ; magic byte sequence end
>  jmp far [cs:.next]
> .handle:
>  cmp al, 0
>  ; check if installation check
>  mov al, 0
>  ; prepare "function not supported" return code
>  jne .iret
>  ; jump to iret if not installation check -->
>  ; (flags still as set from the compare instruction)
>  mov al, 0FFh
>  mov cx, 0100h
>  mov di, amis_signature
>  mov dx, cs
> .iret:
>  iret
> 
> 
> ETA: I found that you actually already uploaded your work! Nicely done. 
> Here's what I am referring to [2].
> 
> No need to reserve 15 bytes for the AMIS signature description, just let the 
> assembler do its job and allocate exactly as many bytes as needed.

I think I might drop the “optional” description completely to shave off 14 
bytes.

> The multiplex number, as in my example above, can be embedded into the code 
> of the handler. (I do not call this the "multiplexer" as you do here, rather, 
> "multiplexer" in my use refers to the resident program. The number is just 
> the "multiplex number".) This is what both Ralf Brown's examples and my 
> resident programs do.

Once set, it never changes. Great idea to save some bytes. :-)

> If you do put the multiplex number there at the end of the signatures, it may 
> save (very little) space to initialise it as 

Re: [Freedos-devel] Extension proposal

[Bret Johnson]

> For fun, I implemented initial versions of both the 0x2b interrupt
> hook and a partial 0x2d implementation to locate the Logger device
> driver. 
>
> The 0x2d version is very bare bones and does not include several
> functions required to be fully compliant. It only responds to the
> install check function for whichever multiplexer it allocates. Other
> function requests only respond with AL=0x00 which i guess is not
> implemented. 
>
> As such, 0x2d uses 35 bytes more resident memory than 0x2b.
> (actually 49 bytes more if you count a title string that is not
> required). If I spend the time making it fully compliant, it will
> require even more. It really would need function 04 (determine
> chained ints) and should have 06 (device driver info). 

While keeping the memory footprint as small as possible is important, you also 
need to weigh that against the user experience.  To give the user things that 
are useful (even if they don't realize they are useful) usually takes more 
memory than a bare-bones approach, but the extra memory is usually worth the 
trouble.  Sometimes it's worth an extra few hundred bytes to make things easier 
for the user (and for other programs).

That's why I encourage AMIS (which includes IISP) and use it in all new 
versions of my programs.  In early versions I used INT 2Fh.  According to MS, 
INT 2Fh multiplex numbers below C0h are reserved for MS applications but user 
applications can use C0h+.  The problem is, there really is no standard on how 
INT 2Fh should be implemented, even for something as simple as a basic 
installation check.  Even the MS applications aren't consistent between each 
other.

I finally gave up on that idea and there were two viable alternatives that I 
found: AMIS and TesSeRact, with AMIS being vastly superior (though still not 
"perfect").  Even if you end up not using AMIS, I would still recommend IISP.  
Among other things, it lets the user be much less concerned about the order of 
TSR installation, particularly when uninstalling.  AMIS has a lot of other nice 
"features" as well.  I also would not suggest implementing an "incomplete" AMIS.

> I think overall, it might be better to figure out why walking the
> device driver chain was not working correctly. Most likely, it was
> some dumb thing I was doing wrong. It would have the smallest
> resident footprint of the locating schemes.

The device driver chain search usually works pretty well, but doesn't 
necessarily work correctly in all VMs and may not work correctly in early DOS 
"clones", either.  I know that in DOSBox (and its clones/forks) the device 
driver chain is "messed up".  A standard DOS Device Driver chain includes 
several character devices (CON, NUL, COM1-COM4, LPT1-LPT3, AUX, PRN, and 
CLOCK$) but DOSBox only has CON and NUL.  The other devices (at least can) 
exist but are "virtualized" somehow and don't show up in the chain.  BTW, 
that's another test I've added to ISLOADED for the detection of a Virtual 
Machine (or at least a virtual DOS).

> I'll probably take another look at that. Or maybe, I'll just stick
> with the 0x2d multiplexerf despite it's larger footprint. 
>
> All are better (and much faster) than the temporary brute force
> search in previous versions.


___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[C. Masloch]

(Replying to multiple mails again.)

On at 2023-04-21 19:03 -0400, jer...@shidel.net wrote:

Hi Tom,


On Apr 21, 2023, at 2:01 PM, tom ehlert  wrote:

Hi ecm,



[1]: 
https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L267


this got me looking into this 'too slow' detection method.
and it is indeed slow. as in molasse. let me explain.


a) isn't

  %%Comparing:
inc di
lodsb
cmp al, [es:di]
jne %%Next
loop%%Comparing

more or less the definition of

repe cmpsw

??


Yes, more or less it was a  “repe cmpsb”

That was a while ago and I forget the reasoning I did it that way. Possibly, 
not requiring a case-specific match.


b) decompiling the actual code, it's basically


  for (seg = 10; seg < 0xa000; seg++)
{
if (fmemcmp(MK_FP(seg, 9), "LOGGER", DriverLength))
{
return found_our_driver_at(seg);
}
}
   return failure;

that's indeed slow as it compares all memory up to 0xa000 to lookup the driver,
or up to the drivers address (which is much better most of the time.


Yup it is a very slow way to locate the driver. It was only meant to be 
temporary until better method was implemented.

The better method will most likely be INT 0x2D (AMIS).

During the development stage, locating the driver needed to be done quickly so 
other things could be developed and tested.

That process went something like this:

1) We could walk the MCB chain…. That will require some overhead and 
complexity. Too big of a pain for now (see below).


Commented by me below.


2) We could walk the device driver chain. Thats fairly straight forward and 
easy enough to implement. Lets do that… Hmmm, not all device drivers are 
showing up in the chain and logger is not being seen. Let’s not worry about 
that for now and do something else.


Don't know how you managed to do that, it is indeed straightforward.


3) We could hook an interrupt somewhere. Yeah, that will be good and reliable. 
Lets do that. But, which one will not cause any conflicts or collisions. Hmmm, 
lets worry about it later and just get something that works for now.
4) Well a brute force search will work. It is slower than a glacier and as 
clever as a stone. But, it will work well enough that I can get to work on 
writing and testing the important stuff.
5) Brute force search it is then with a very few optimizations… For now, good 
enough.

Even though the delay incurred when launching the interface program is not very 
noticable, it is way to slow.

Now that the important stuff has been written and is being tested, the brute 
force search needs to go.


when I mentioned microseconds, I had the DOS memory chain in mind where you 
would have
to compare 20-50 locations to your drivers name.


I did consider walking the MCB chain to find it. But, that comes with its own 
set of problems. Some blocks contain sub-blocks.


SD "system data" blocks with sub-MCBs are a fact on most kernels, but 
they're not terribly complicated to detect and handle. First two name 
bytes are "SD" and the owner is 8. Sub-MCBs have signature letters 
indicating the type of the block rather than "M" link / "Z" no link, and 
they always span the entire space up to the end of the container SD block.



The upper memory blocks may or may not be linked into the primary MCB chain.


This is more of a problem. You can ignore a "Z" (ie, treat it as if it 
is an "M") if you know that the next block after the "Z" block is the 
first UMCB. You can find the first UMCB by trying to call interrupt 2Fh, 
function 1261h, CY; if it returns NC then ax = first UMCB. Or if the 
kernel does not support that function, switch off the UMB link, walk 
until "Z", switch on the UMB link, walk until you're past the MCB that 
was "Z" prior to this (but is now "M"). Of course, you want to preserve 
the original UMB link state around this. Example code is in my TSR 
example [1].


For a single search, it may be more efficient to just do the UMB link 
state enabling around your MCB walker as is, and honor any "Z" as the 
final end marker then.



[1]: https://hg.pushbx.org/ecm/tsr/file/749e0f25364c/transien.asm#l96



There are other aspects involving interaction between the Driver, Interface and 
Log that are also just “good enough for now” and will probably be changed a 
great deal.

It is an alpha version for a reason.

:-)



On at 2023-04-23 19:17 -0400, jer...@shidel.net wrote:
For “fun”, I implemented initial versions of both the 0x2b interrupt hook and a partial 0x2d implementation to locate the Logger device driver. 


The 0x2d version is very bare bones and does not include several functions 
required to be “fully compliant”. It only responds to the install check 
function for whichever multiplexer it allocates. Other function requests only 
respond with AL=0x00 which i guess is “not 

Re: [Freedos-devel] Extension proposal

[jerome]

Hi All,

For “fun”, I implemented initial versions of both the 0x2b interrupt hook and a 
partial 0x2d implementation to locate the Logger device driver. 

The 0x2d version is very bare bones and does not include several functions 
required to be “fully compliant”. It only responds to the install check 
function for whichever multiplexer it allocates. Other function requests only 
respond with AL=0x00 which i guess is “not implemented”. 

As such, 0x2d uses 35 bytes more resident memory than 0x2b. (actually 49 bytes 
more if you count a “title” string that is not required). If I spend the time 
making it fully compliant, it will require even more. It really would need 
function 04 (determine chained ints) and should have 06 (device driver info). 

I think overall, it might be better to figure out why walking the device driver 
chain was not working correctly. Most likely, it was some dumb thing I was 
doing wrong. It would have the smallest resident footprint of the locating 
schemes. 

I’ll probably take another look at that. Or maybe, I’ll just stick with the 
0x2d multiplexer despite it’s larger footprint. 

All are better (and much faster) than the temporary brute force search in 
previous versions.

:-)

Jerome

___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[jerome]

Hi Tom,

> On Apr 21, 2023, at 2:01 PM, tom ehlert  wrote:
> 
> Hi ecm,
> 
> 
>> [1]: 
>> https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L267
> 
> this got me looking into this 'too slow' detection method.
> and it is indeed slow. as in molasse. let me explain.
> 
> 
> a) isn't
> 
>  %%Comparing:
>inc di
>lodsb
>cmp al, [es:di]
>jne %%Next
>loop%%Comparing
> 
> more or less the definition of
> 
>repe cmpsw
> 
> ??

Yes, more or less it was a  “repe cmpsb”

That was a while ago and I forget the reasoning I did it that way. Possibly, 
not requiring a case-specific match. 

> b) decompiling the actual code, it's basically
> 
> 
>  for (seg = 10; seg < 0xa000; seg++)
>{
>if (fmemcmp(MK_FP(seg, 9), "LOGGER", DriverLength))
>{
>return found_our_driver_at(seg);
>}
>}
>   return failure;
> 
> that's indeed slow as it compares all memory up to 0xa000 to lookup the 
> driver,
> or up to the drivers address (which is much better most of the time.

Yup it is a very slow way to locate the driver. It was only meant to be 
temporary until better method was implemented.  

The better method will most likely be INT 0x2D (AMIS).

During the development stage, locating the driver needed to be done quickly so 
other things could be developed and tested. 

That process went something like this:

1) We could walk the MCB chain…. That will require some overhead and 
complexity. Too big of a pain for now (see below).
2) We could walk the device driver chain. Thats fairly straight forward and 
easy enough to implement. Lets do that… Hmmm, not all device drivers are 
showing up in the chain and logger is not being seen. Let’s not worry about 
that for now and do something else.
3) We could hook an interrupt somewhere. Yeah, that will be good and reliable. 
Lets do that. But, which one will not cause any conflicts or collisions. Hmmm, 
lets worry about it later and just get something that works for now.
4) Well a brute force search will work. It is slower than a glacier and as 
clever as a stone. But, it will work well enough that I can get to work on 
writing and testing the important stuff. 
5) Brute force search it is then with a very few optimizations… For now, good 
enough. 

Even though the delay incurred when launching the interface program is not very 
noticable, it is way to slow.

Now that the important stuff has been written and is being tested, the brute 
force search needs to go. 

> when I mentioned microseconds, I had the DOS memory chain in mind where you 
> would have 
> to compare 20-50 locations to your drivers name.

I did consider walking the MCB chain to find it. But, that comes with its own 
set of problems. Some blocks contain sub-blocks. The upper memory blocks may or 
may not be linked into the primary MCB chain. 

There are other aspects involving interaction between the Driver, Interface and 
Log that are also just “good enough for now” and will probably be changed a 
great deal. 

It is an alpha version for a reason.

:-)

Jerome




___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[jerome]

Hi,

> On Apr 21, 2023, at 12:43 PM, C. Masloch  wrote:
> 
> I'm responding to all of your mails (Jerome's) so far.
> 
> On at 2023-04-21 06:55 -0400, jer...@shidel.net wrote:
>> Hi All,
>> At present, the interface program for Logger just performs a slightly 
>> optimized brute force search for the Logger device driver. Although 
>> reliable, it is very slow compared to providing a simple interrupt call to 
>> test for installation.
> 
> I agree that scanning memory is not good enough. However, you could find the 
> DOS device driver chain and scan it for your driver. It starts with the "NUL" 
> device header in the DOS data segment (get approximate location with 
> interrupt 21h function 52h) and ends when an offset of the "next device" 
> pointer is 0h.
> 
> I also examined your implementation so far [1]. Here's some things I noticed:
> 
> 1. You misspellt "it's location", should be "its".
> 2. You start the search at paragraph 0100h. It is a remote possibility for 
> the initial low memory area part of DOS to take up less than 4096 bytes 
> however.
> 3. Your %%Comparing loop can use lodsb \ scasb or even cmpsb in fact.
> 4. If SEARCH_LOW is defined then when bx is equal to dx (your .COM 
> application's ds) and dx was above 0A000h it will loop back to %%Next, 
> incrementing bx again. I don't understand the logic behind this. If bx was 
> below-or-equal 0A000h then you set it to 0A000h here. I'm fairly sure this is 
> not very good, because your application can load lower than the device driver 
> but the latter can still be below 0A000h.
> 5. push bx \ pop es is not very useful when mov es, bx is also 2 bytes.
> 6. A comment at %%Done says that upon reaching there with CY, es equals 
> "undefined (0x)". This is only true for when the SEARCH_LOW option is in 
> use. Otherwise it might end at 9FFFh, I think.
> 7. It is confusing to initialise di to 9 for the segment to check then have 
> it immediately incremented to 10 (the offset of the device driver name in the 
> device header) in the first iteration of %%Comparing. Could do with a 
> comment, at least.

Yeah, that is just temporary function to locate the driver. If it was going to 
stick with a brute force search, it could use optimized more and other fixes.

But it would be better to just rip that section out for a better install check 
procedure. Like the code comment says, “Not perfect or fast. But, good enough 
for now.”

I was starting to improve it after getting your reply. But, it is better to 
just replace it now.

> 
> The CMPDD macro (just above what I linked in [1]) is neat to be sure. The 
> conditional code mechanism [2] also looks good.
> 
> 
> [1]: 
> https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L267
> [2]: 
> https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L1102
> 
> 
>> Looking at the different interrupts, I think I have come up with a solution 
>> that will work well for Logger and any other driver or program that needs 
>> such a check. So, I’d like to propose a “standard” we could use. I’d like to 
>> get your feedback on what I’m thinking…
> 
> As the others have mentioned, there is a "standard" already. AMIS provides 
> everything you propose, and more. And if you think you don't want to support 
> all of the bits, even those required for specification compliance, you can 
> always pick and choose what to include.
> 
>> Looking at RBIL, interrupt 0x2b is barely used by anything. Under MS-DOS and 
>> FreeDOS, this simply points to an IRET. Under IBM ROM-DOS, AH functions 
>> 0x00-0x03 do some things. But, all other calls do nothing.
>> https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html 
>> 
> 
> This is precisely the rationale of why AMIS (eventually) ended up on 
> interrupt 2Dh. It is not generally in use, and points to an iret on MS-DOS v3 
> compatibles by default. I'm about to add a check for its validity to my TSR 
> applications, which checks that the segment is nonzero and the offset is not 
> equal to 0h [3]. (This check is only for the example for now, but I will 
> pick it into all my TSRs eventually.)
> 
> 
> [3]: https://hg.pushbx.org/ecm/tsr/rev/aafe8caaf4f5
> 
> 
>> An install check issuing this interrupt would be simple to perform. A 
>> program could set the Carry Flag,
> 
> Carry Flag is a nonstarter for a multiplex interrupt check, because you 
> either need to return with retf 2 (likely scrambling the Direction Flag and 
> Trace Flag and Interrupt Flag), or do a complicated dance to pass the CF or 
> most arithmetic status flags to the fl word on the interrupt stack frame, eg 
> using lahf [4] [5] (arithmetic flags except Overflow Flag) or a little rcr \ 
> rol trick [6] (modifies Carry Flag only).
> 
> 
> [4]: https://github.com/640-KB/GLaBIOS/issues/20
> [5]: https://github.com/MobyGamer/softhddi/issues/1
> [6]: 

Re: [Freedos-devel] Extension proposal

[tom ehlert]

Hi ecm,


> [1]: 
> https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L267

this got me looking into this 'too slow' detection method.
and it is indeed slow. as in molasse. let me explain.


a) isn't

  %%Comparing:
inc di
lodsb
cmp al, [es:di]
jne %%Next
loop%%Comparing

more or less the definition of

repe cmpsw

??


b) decompiling the actual code, it's basically


  for (seg = 10; seg < 0xa000; seg++)
{
if (fmemcmp(MK_FP(seg, 9), "LOGGER", DriverLength))
{
return found_our_driver_at(seg);
}
}
   return failure;

that's indeed slow as it compares all memory up to 0xa000 to lookup the driver,
or up to the drivers address (which is much better most of the time.

when I mentioned microseconds, I had the DOS memory chain in mind where you 
would have 
to compare 20-50 locations to your drivers name.









___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[C. Masloch]

I'm responding to all of your mails (Jerome's) so far.

On at 2023-04-21 06:55 -0400, jer...@shidel.net wrote:

Hi All,

At present, the interface program for Logger just performs a slightly 
optimized brute force search for the Logger device driver. Although 
reliable, it is very slow compared to providing a simple interrupt call 
to test for installation.


I agree that scanning memory is not good enough. However, you could find 
the DOS device driver chain and scan it for your driver. It starts with 
the "NUL" device header in the DOS data segment (get approximate 
location with interrupt 21h function 52h) and ends when an offset of the 
"next device" pointer is 0h.


I also examined your implementation so far [1]. Here's some things I 
noticed:


1. You misspellt "it's location", should be "its".
2. You start the search at paragraph 0100h. It is a remote possibility 
for the initial low memory area part of DOS to take up less than 4096 
bytes however.

3. Your %%Comparing loop can use lodsb \ scasb or even cmpsb in fact.
4. If SEARCH_LOW is defined then when bx is equal to dx (your .COM 
application's ds) and dx was above 0A000h it will loop back to %%Next, 
incrementing bx again. I don't understand the logic behind this. If bx 
was below-or-equal 0A000h then you set it to 0A000h here. I'm fairly 
sure this is not very good, because your application can load lower than 
the device driver but the latter can still be below 0A000h.

5. push bx \ pop es is not very useful when mov es, bx is also 2 bytes.
6. A comment at %%Done says that upon reaching there with CY, es equals 
"undefined (0x)". This is only true for when the SEARCH_LOW option 
is in use. Otherwise it might end at 9FFFh, I think.
7. It is confusing to initialise di to 9 for the segment to check then 
have it immediately incremented to 10 (the offset of the device driver 
name in the device header) in the first iteration of %%Comparing. Could 
do with a comment, at least.


The CMPDD macro (just above what I linked in [1]) is neat to be sure. 
The conditional code mechanism [2] also looks good.



[1]: 
https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L267
[2]: 
https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L1102



Looking at the different interrupts, I think I 
have come up with a solution that will work well for Logger and any 
other driver or program that needs such a check. So, I’d like to propose 
a “standard” we could use. I’d like to get your feedback on what I’m 
thinking…


As the others have mentioned, there is a "standard" already. AMIS 
provides everything you propose, and more. And if you think you don't 
want to support all of the bits, even those required for specification 
compliance, you can always pick and choose what to include.


Looking at RBIL, interrupt 0x2b is barely used by anything. Under MS-DOS 
and FreeDOS, this simply points to an IRET. Under IBM ROM-DOS, AH 
functions 0x00-0x03 do some things. But, all other calls do nothing.


https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html 



This is precisely the rationale of why AMIS (eventually) ended up on 
interrupt 2Dh. It is not generally in use, and points to an iret on 
MS-DOS v3 compatibles by default. I'm about to add a check for its 
validity to my TSR applications, which checks that the segment is 
nonzero and the offset is not equal to 0h [3]. (This check is only 
for the example for now, but I will pick it into all my TSRs eventually.)



[3]: https://hg.pushbx.org/ecm/tsr/rev/aafe8caaf4f5


An install check issuing this interrupt would be simple to perform. A 
program could set the Carry Flag,


Carry Flag is a nonstarter for a multiplex interrupt check, because you 
either need to return with retf 2 (likely scrambling the Direction Flag 
and Trace Flag and Interrupt Flag), or do a complicated dance to pass 
the CF or most arithmetic status flags to the fl word on the interrupt 
stack frame, eg using lahf [4] [5] (arithmetic flags except Overflow 
Flag) or a little rcr \ rol trick [6] (modifies Carry Flag only).



[4]: https://github.com/640-KB/GLaBIOS/issues/20
[5]: https://github.com/MobyGamer/softhddi/issues/1
[6]: https://hg.pushbx.org/ecm/seekext/file/bbfcfa0d1c0b/resident.asm#l254


load AH/AX with “check for install” 
function and set DS:BX to point to an identifier string (minimum 8 
characters, no maximum). Then call the interrupt.


AMIS already provides a standard way to do this. Instead of providing a 
pointer to the resident handlers, however, it returns a pointer to the 
resident's signatures. Those take up at least 17 bytes, 8 bytes intended 
for "manufacturer", the next 8 bytes for "product", and then up to 64 
bytes for a longer descriptive name (NUL-terminated). This longer string 
is optional in that it can just hold the empty string, and in that case 
you 

Re: [Freedos-devel] Extension proposal

[jerome]

Hi Eric, 

> On Apr 21, 2023, at 10:27 AM, Eric Auer  wrote:
> 
> 
> Hi Jerome,
> 
> if you are worried about collisions, use AMIS int 2d.
> Overhead is low and it is less crowded than MUX int 2f.
> 
> RBIL was famous enough to make AMIS more widespread. FreeDOS is too
> niche to re-invent that wheel and declare int 2b to be a new trend.
> 
> Why would your RESIDENT driver have to do install checks?
> You can do those in the TRANSIENT part and just store the
> results as static data in the resident driver part.

There driver does do an install check at start up to prevent multiple 
instances. 
However, all of that code is discarded after initialization. But, you still 
need to 
“store the results” in the resident portion and respond to the interrupt calls
properly. 

There is probably little difference in the resident portion between using 2D or 
2B. Only, writing both versions could say for certain which is larger and by
how much. But, RBIL says 64-bytes + 22 per hooked interrupt. A simple ID
string comparison and set return values won't use much.

But we are taking about bytes and the savings may not be worth it.

> So the risk of wasting RAM by using AMIS are minimal :-)
> 
> I agree that MUX can get crowded. However, for disk I/O,
> the actual I/O still is the main bottleneck (unless you
> are a large cache with a very inefficient algorithm) and
> you would not do install checks for each invocation anyway.
> 
> One example of slow call chains which you can feel may
> be old ANSI variants and slow system and VGA BIOS. But
> that was long ago and even those were not THAT slow.
> 
> AMIS can be expected to be much faster than MUX and it
> will not be in the way for your CDEX performance either.
> 
> If you want to avoid the hassle of scanning for a free
> magic number, I would just hardcode one. Collision risk
> will still be lower than when grabbing the whole int 2b.
> 
> I agree with Tom: Use int 2d, it is the better choice and
> 2023 is not the year for MORE standards for simple things.

The only advantages I see in using 2B instead are as follows, it is 
very light weight, it uses very little memory to implement, it has 
almost no chance of conflicts and only being used for an install check
there is no impact on performance of any aspect of the system. 

Unless I just stay with the brute force search, 2D is probably the 
better choice. There are even suggestions that memory managers
can query it to retrieve the driver name. Although, I don’t know 
if the FreeDOS mem command makes use of that or not.

> Regards, Eric

:-)

Jerome

> ___
> Freedos-devel mailing list
> Freedos-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-devel



___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[jerome]

> On Apr 21, 2023, at 10:21 AM, tom ehlert  wrote:
> 
> Hi,
> 
> 
>> At present, the interface program for Logger just performs a slightly 
>> optimized brute force search for the Logger device driver. Although 
>> reliable, it is very slow compared to providing a simple interrupt call to 
>> test for installation.
> 
> given that this detection will be done once per program start: how many 
> microseconds do you expect to save?

Well, there is the catch.

At present, the current semi-optimized brute force search is quick. The delay 
in finding the driver is not really noticeable when a user invokes the 
interface program to view, print or perform some other operation. 

But there is a delay and those delays can add up. For example, if the FreeDOS 
installer ran the interface program many times to put messages into the log or 
write the results of programs like FDISK there for debugging installation 
issues. All those executions will add up. 

But, like you said, how much performance will be gained? For example, if adding 
a comment to the log takes 1 second to load the interface program from floppy 
diskette and only 1/1000 of a second to find the driver, then improving that 
makes almost no sense at all.  

I should probably benchmark it to see if it is worth the bother. But, it would 
vary widely based on system and numerous other factors.

> 
>> Looking at the different interrupts, I think I have come up with a solution 
>> that will work well for Logger and any other driver or program that needs 
>> such a check. So, I’d like to propose a “standard” we could use. I’d like to 
>> get your feedback on what I’m thinking…
> 
> setting a "standard" which is used probably exactly once ? we write year 2023 
> ;)
> 
> https://xkcd.com/927/
> 
> INT 2D has been mentioned by others

Maybe go for 16 instead. It is an even binary number. LOL

XKCD is great. 

> 
> 
>> Looking at RBIL, interrupt 0x2b is barely used by anything. Under MS-DOS and 
>> FreeDOS, this simply points to an IRET. Under IBM ROM-DOS, AH functions 
>> 0x00-0x03 do some things. But, all other calls do nothing. 
> 
>> https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html 
>> 
> 
>> An install check issuing this interrupt would be simple to perform. A 
>> program could set the Carry Flag, load AH/AX with “check for install” 
>> function and set DS:BX to point to an identifier string (minimum 8 
>> characters, no maximum). Then call the interrupt. On return, if the Carry 
>> Flag is still set, then the install check failed. If the Carry Flag is 
>> clear, it succeeded and other register would be modified to according to the 
>> needs of the programs. 
> 
>> Implementation for Logger (as an example) could be:
> 
>> 
>> CHECK:
>>push cs
>>pop  ds ; set DS to our code segment
>>stc ; set the carry flag
>>mov  ax, 0xfd00 ; set AX to install check function
>>mov  bx, LOGGER_ID  ; offset to LOGGER device driver ID
>>int  0x2b   ; call install check interrupt
>>jc   NOT_INSTALLED  ; nothing changed, driver is not 
>> loaded
> 
>   ;you should add 
>  cmp ax, MAGIC_VALUE
>  jne   NOT_INSTALLED
> 
> your proposed INT2B might be used by some other software that for some crazy 
> reasons 
> clears the carry flag.

Your correct of course. :-)

In implementation, I would probably have it return the pointer to the driver 
header and verify the signature of the driver again by the caller. 

> 
> anyway, INT2D is probably the better choice anyway.

Probably, I just don’t like to add extra code and increase the resident 
footprint without good cause.

:-)

Jerome


> 
> 
> ___
> Freedos-devel mailing list
> Freedos-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-devel



___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[Eric Auer]

Hi Jerome,

if you are worried about collisions, use AMIS int 2d.
Overhead is low and it is less crowded than MUX int 2f.

RBIL was famous enough to make AMIS more widespread. FreeDOS is too
niche to re-invent that wheel and declare int 2b to be a new trend.

Why would your RESIDENT driver have to do install checks?
You can do those in the TRANSIENT part and just store the
results as static data in the resident driver part.

So the risk of wasting RAM by using AMIS are minimal :-)

I agree that MUX can get crowded. However, for disk I/O,
the actual I/O still is the main bottleneck (unless you
are a large cache with a very inefficient algorithm) and
you would not do install checks for each invocation anyway.

One example of slow call chains which you can feel may
be old ANSI variants and slow system and VGA BIOS. But
that was long ago and even those were not THAT slow.

AMIS can be expected to be much faster than MUX and it
will not be in the way for your CDEX performance either.

If you want to avoid the hassle of scanning for a free
magic number, I would just hardcode one. Collision risk
will still be lower than when grabbing the whole int 2b.

I agree with Tom: Use int 2d, it is the better choice and
2023 is not the year for MORE standards for simple things.

Regards, Eric




___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[tom ehlert]

Hi,


> At present, the interface program for Logger just performs a slightly 
> optimized brute force search for the Logger device driver. Although reliable, 
> it is very slow compared to providing a simple interrupt call to test for 
> installation.

given that this detection will be done once per program start: how many 
microseconds do you expect to save?

>  Looking at the different interrupts, I think I have come up with a solution 
> that will work well for Logger and any other driver or program that needs 
> such a check. So, I’d like to propose a “standard” we could use. I’d like to 
> get your feedback on what I’m thinking…

setting a "standard" which is used probably exactly once ? we write year 2023 ;)

https://xkcd.com/927/

INT 2D has been mentioned by others



> Looking at RBIL, interrupt 0x2b is barely used by anything. Under MS-DOS and 
> FreeDOS, this simply points to an IRET. Under IBM ROM-DOS, AH functions 
> 0x00-0x03 do some things. But, all other calls do nothing. 

>  https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html 
> 

> An install check issuing this interrupt would be simple to perform. A program 
> could set the Carry Flag, load AH/AX with “check for install” function and 
> set DS:BX to point to an identifier string (minimum 8 characters, no 
> maximum). Then call the interrupt. On return, if the Carry Flag is still set, 
> then the install check failed. If the Carry Flag is clear, it succeeded and 
> other register would be modified to according to the needs of the programs. 

> Implementation for Logger (as an example) could be:

> 
> CHECK:
> push cs
> pop  ds ; set DS to our code segment
> stc ; set the carry flag
> mov  ax, 0xfd00 ; set AX to install check function
> mov  bx, LOGGER_ID  ; offset to LOGGER device driver ID
> int  0x2b   ; call install check interrupt
> jc   NOT_INSTALLED  ; nothing changed, driver is not 
> loaded

   ;you should add 
  cmp ax, MAGIC_VALUE
  jne   NOT_INSTALLED

your proposed INT2B might be used by some other software that for some crazy 
reasons 
clears the carry flag.

anyway, INT2D is probably the better choice anyway.



___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[jerome]

Hi Eric and Bret,

I looked at 2D, 2F and even absolute AMIS 7D. 

> On Apr 21, 2023, at 7:44 AM, Eric Auer  wrote:
> 
> 
> Hi Jerome,
> 
> there is no need to allocate a whole int 0x2b for just one driver.

It need not be just for one driver. 

Anyone could hook 0x2b and is a very loose and light weight method with an 
extremely low possibility of collision or conflict. Also, nothing else uses 2B 
so it will not effect general system performance. We also don’t need to worry 
about Microsoft changing things and breaking it. 

> There are mechanisms which already invite drivers to share them :-)
> 
> INT 2D - ALTERNATE MULTIPLEX INTERRUPT SPECIFICATION (AMIS) [v3.6]
>AH = multiplex number
>AL = function
>00h installation check
>01h get private entry point
>02h uninstall
>03h request popup
>04h determine chained interrupts
>05h get hotkey list
>06h get device-driver information
>07h-0Fh reserved for future enhancements
>Return: AL = 00h (not implemented)
>other  application-dependent
>other registers vary by function (also see individual entries below)
> 
> You also are in good company there, for example screen thief uses it.

https://fd.lod.bz/rbil/interrup/tsr/2d.html 


I was looking at it. It seems to require a good deal of additional overhead and 
complexity for just a “hey, where are you?” install check. Which will increase 
the memory resident footprint a little unnecessarily. 

However, it is an (at least partially) accepted “standard” and possibly the 
better solution and would not increase the footprint that much. It is 
definitely worth further consideration.

> 
> INT 2F - Multiplex - NOTES
>AH = identifier of program which is to handle the interrupt
>   00h-3Fh reserved for IBM (for DOS)
>   40h-7Fh reserved for Microsoft (for DOS)
>   80h-B7h reserved for IBM
>   B8h-BFh reserved for networks
>   C0h-FFh reserved for applications
>AL is the function code
>   This is a general mechanism for verifying the presence of a TSR and
>   communicating with it.
> 
> AMIS got introduced (by RBIL, sort of?) because so many apps use INT 2F,
> but if you only need an install check, performance is no issue and you
> can use INT 2F without problems.
> 
> You can also use INT 2F just for detection and then acquire a pointer
> to a fast call provided by your driver if you need something quick.

I don’t think 2F would be a good choice. In part, you have the network 
redirector and other filesystem related functions on that interrupt. Every hook 
that gets installed there takes a little slice of time to process. Which in 
turn, can slow down normal file operations. 

Also, I think that there is to high a risk of having a possible collision with 
2F with programs that just arbitrarily grab a fixed “unused” function to 
intercept.

> 
> Regards, Eric
> 

I think it comes down to either using 2D or 2B. I like 2B because there is 
almost no overhead and very simple to implement on both ends. But, 2D might be 
the better way to go and is a pre-existing semi-accepted standard.

:-)

Jerome

> 
> 
> ___
> Freedos-devel mailing list
> Freedos-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-devel

___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[Bret Johnson]

Why not use AMIS (INT 2Dh)?  It's there for the very reason you're proposing.  
However, it's much more involved than your simple test, but also provides many, 
many additional features and allows simple communication, interaction, and even 
cooperation between all TSRs and Device Drivers that support it.


___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Extension proposal

[Eric Auer]

Hi Jerome,

there is no need to allocate a whole int 0x2b for just one driver.
There are mechanisms which already invite drivers to share them :-)

INT 2D - ALTERNATE MULTIPLEX INTERRUPT SPECIFICATION (AMIS) [v3.6]
AH = multiplex number
AL = function
00h installation check
01h get private entry point
02h uninstall
03h request popup
04h determine chained interrupts
05h get hotkey list
06h get device-driver information
07h-0Fh reserved for future enhancements
Return: AL = 00h (not implemented)
other  application-dependent
other registers vary by function (also see individual entries 
below)


You also are in good company there, for example screen thief uses it.

INT 2F - Multiplex - NOTES
AH = identifier of program which is to handle the interrupt
   00h-3Fh reserved for IBM (for DOS)
   40h-7Fh reserved for Microsoft (for DOS)
   80h-B7h reserved for IBM
   B8h-BFh reserved for networks
   C0h-FFh reserved for applications
AL is the function code
   This is a general mechanism for verifying the presence of a TSR and
   communicating with it.

AMIS got introduced (by RBIL, sort of?) because so many apps use INT 2F,
but if you only need an install check, performance is no issue and you
can use INT 2F without problems.

You can also use INT 2F just for detection and then acquire a pointer
to a fast call provided by your driver if you need something quick.

Regards, Eric




___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel