I'm responding to all of your mails (Jerome's) so far.
On at 2023-04-21 06:55 -0400, jer...@shidel.net wrote:
Hi All,
At present, the interface program for Logger just performs a slightly
optimized brute force search for the Logger device driver. Although
reliable, it is very slow compared to providing a simple interrupt call
to test for installation.
I agree that scanning memory is not good enough. However, you could find
the DOS device driver chain and scan it for your driver. It starts with
the "NUL" device header in the DOS data segment (get approximate
location with interrupt 21h function 52h) and ends when an offset of the
"next device" pointer is 0FFFFh.
I also examined your implementation so far [1]. Here's some things I
noticed:
1. You misspellt "it's location", should be "its".
2. You start the search at paragraph 0100h. It is a remote possibility
for the initial low memory area part of DOS to take up less than 4096
bytes however.
3. Your %%Comparing loop can use lodsb \ scasb or even cmpsb in fact.
4. If SEARCH_LOW is defined then when bx is equal to dx (your .COM
application's ds) and dx was above 0A000h it will loop back to %%Next,
incrementing bx again. I don't understand the logic behind this. If bx
was below-or-equal 0A000h then you set it to 0A000h here. I'm fairly
sure this is not very good, because your application can load lower than
the device driver but the latter can still be below 0A000h.
5. push bx \ pop es is not very useful when mov es, bx is also 2 bytes.
6. A comment at %%Done says that upon reaching there with CY, es equals
"undefined (0xffff)". This is only true for when the SEARCH_LOW option
is in use. Otherwise it might end at 9FFFh, I think.
7. It is confusing to initialise di to 9 for the segment to check then
have it immediately incremented to 10 (the offset of the device driver
name in the device header) in the first iteration of %%Comparing. Could
do with a comment, at least.
The CMPDD macro (just above what I linked in [1]) is neat to be sure.
The conditional code mechanism [2] also looks good.
[1]:
https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L267
[2]:
https://gitlab.com/DOSx86/logger/-/blob/aae3dfddcdacfea18950a96ce9449767c20b2d66/source/logger/common.inc#L1102
Looking at the different interrupts, I think I
have come up with a solution that will work well for Logger and any
other driver or program that needs such a check. So, I’d like to propose
a “standard” we could use. I’d like to get your feedback on what I’m
thinking…
As the others have mentioned, there is a "standard" already. AMIS
provides everything you propose, and more. And if you think you don't
want to support all of the bits, even those required for specification
compliance, you can always pick and choose what to include.
Looking at RBIL, interrupt 0x2b is barely used by anything. Under MS-DOS
and FreeDOS, this simply points to an IRET. Under IBM ROM-DOS, AH
functions 0x00-0x03 do some things. But, all other calls do nothing.
https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html
<https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html>
This is precisely the rationale of why AMIS (eventually) ended up on
interrupt 2Dh. It is not generally in use, and points to an iret on
MS-DOS v3 compatibles by default. I'm about to add a check for its
validity to my TSR applications, which checks that the segment is
nonzero and the offset is not equal to 0FFFFh [3]. (This check is only
for the example for now, but I will pick it into all my TSRs eventually.)
[3]: https://hg.pushbx.org/ecm/tsr/rev/aafe8caaf4f5
An install check issuing this interrupt would be simple to perform. A
program could set the Carry Flag,
Carry Flag is a nonstarter for a multiplex interrupt check, because you
either need to return with retf 2 (likely scrambling the Direction Flag
and Trace Flag and Interrupt Flag), or do a complicated dance to pass
the CF or most arithmetic status flags to the fl word on the interrupt
stack frame, eg using lahf [4] [5] (arithmetic flags except Overflow
Flag) or a little rcr \ rol trick [6] (modifies Carry Flag only).
[4]: https://github.com/640-KB/GLaBIOS/issues/20
[5]: https://github.com/MobyGamer/softhddi/issues/1
[6]: https://hg.pushbx.org/ecm/seekext/file/bbfcfa0d1c0b/resident.asm#l254
load AH/AX with “check for install”
function and set DS:BX to point to an identifier string (minimum 8
characters, no maximum). Then call the interrupt.
AMIS already provides a standard way to do this. Instead of providing a
pointer to the resident handlers, however, it returns a pointer to the
resident's signatures. Those take up at least 17 bytes, 8 bytes intended
for "manufacturer", the next 8 bytes for "product", and then up to 64
bytes for a longer descriptive name (NUL-terminated). This longer string
is optional in that it can just hold the empty string, and in that case
you only need to allocate 17 bytes for all the signatures. The code to
support this is about 20 bytes, so we get 37 bytes in total. And that is
with a version code returned in cx.
-a 100
21C9:0100 cmp ah, 26
21C9:0103 je 110
21C9:0105 test al, al
21C9:0107 je 110
21C9:0109 dec al
21C9:010B mov cx, 100
21C9:010E mov di, 100
21C9:0111 mov dx, cs
21C9:0113 iret
21C9:0114
-
This compares to at least 18 bytes of resident code for your proposal,
plus 8 bytes for the signature (= 26 bytes) that you store in the code
segment, and that is with all registers free to change. I'm also not
counting any code required to signal success and return a memory
reference to the caller, which my AMIS example does both include (if you
are willing to dual-use the signature pointer for its intended purpose
plus pointing to your resident program).
-a 100
21C9:0100 cmp ah, 26
21C9:0103 je 110
21C9:0105 push cs
21C9:0106 pop es
21C9:0107 mov di, 100
21C9:010A mov cx, 4
21C9:010D repe cmpsw
21C9:010F je 110
21C9:0111 iret
21C9:0112
-
The AMIS design is also better in using and returning only GPRs for its
installation check function, no segment registers. This helps to use the
interface eg from DPMI applications.
On return, if the
Carry Flag is still set, then the install check failed. If the Carry
Flag is clear, it succeeded and other register would be modified to
according to the needs of the programs.
Implementation for Logger (as an example) could be:
CHECK:
push cs
pop ds; set DS to our code segment
stc; set the carry flag
mov ax, 0xfd00; set AX to install check function
mov bx, LOGGER_ID; offset to LOGGER device driver ID
int 0x2b; call install check interrupt
jc NOT_INSTALLED; nothing changed, driver is not loaded
mov [DRIVER_PTR + 2], es; save segment of far call to driver
function dispatcher
mov [DRIVER_PTR], di; save offset of far call to driver function
dispatcher
jmp MAIN
DRIVER_PTR:
dw 0, 0; pointer to driver function dispatcher
LOGGER_ID:
db 'LOGGERxx'; Logger device driver identifier. 8+ Characters.
MAIN:; do program stuff
NOT_INSTALLED:; driver not loaded, die now with error message
As for the installed Driver, TSR or other extension, you just hook int
0x2b. If AX=0xFD00 and DS:BX points to an identifier you recognize, then
it would clear the carry flag and return values in whatever registers
required. Otherwise, it completely ignore it and just jump to the
previous interrupt handler.
This raises several questions:
1. If it eventually chains the call, should it preserve all register
values from the original caller? If yes, you are looking at
significantly more code than needed by the minimum AMIS multiplexer.
2. Your callout is specific to each program as you provide the signature
rather than letting the multiplexer return its signature.
3. If 2 wasn't already a problem then you still would have an interface
that could modify registers unknown to the caller. Is it enough to
preserve all 16-bit GPRs and segregs? 386 registers? FPU registers, MMX,
XMM, etc?
It requires no additions to the kernel, is backwards compatible and
should have a low probability of any collisions.
Any thoughts on using it as a very minimal “standard” extension
installation check?
:-)
I do have thoughts, and most of them go back to "use a standard that
exists already". The backwards compatibility and kernel agnostic nature
of AMIS is the same as for your proposal. Unlike your proposal so far,
AMIS multiplexers can be scanned by other programs regardless of whether
they "know" the particular multiplexer.
On at 2023-04-21 10:03 -0400, jer...@shidel.net wrote:
> Hi Eric and Bret,
>
> I looked at 2D, 2F and even absolute AMIS 7D.
Interrupt 7Dh was an early AMIS proposal. As far as I am aware it was
never actually implemented.
Interrupt 2Fh has problems like being used for the FS redirector
(function 11h), DOS internal functions often used by the redirector
(function 12h), and the FreeDOS SHARE installation check callout and all
its business logic (function 10h). And there was little agreement what
exactly an installation check should entail, or imply.
>> On Apr 21, 2023, at 7:44 AM, Eric Auer <e.a...@jpberlin.de
>> <mailto:e.a...@jpberlin.de>> wrote:
>>
>>
>> Hi Jerome,
>>
>> there is no need to allocate a whole int 0x2b for just one driver.
>
> It need not be just for one driver.
>
> Anyone could hook 0x2b and is a very loose and light weight method with
> an extremely low possibility of collision or conflict. Also, nothing
> else uses 2B so it will not effect general system performance. We also
> don’t need to worry about Microsoft changing things and breaking it.
All true of AMIS as well.
>> There are mechanisms which already invite drivers to share them :-)
>>
>> INT 2D - ALTERNATE MULTIPLEX INTERRUPT SPECIFICATION (AMIS) [v3.6]
>> AH = multiplex number
>> AL = function
>> 00h installation check
>> 01h get private entry point
>> 02h uninstall
>> 03h request popup
>> 04h determine chained interrupts
>> 05h get hotkey list
>> 06h get device-driver information
>> 07h-0Fh reserved for future enhancements
>> Return: AL = 00h (not implemented)
>> other application-dependent
>> other registers vary by function (also see individual entries
>> below)
>>
>> You also are in good company there, for example screen thief uses it.
>
> https://fd.lod.bz/rbil/interrup/tsr/2d.html
> <https://fd.lod.bz/rbil/interrup/tsr/2d.html>
>
> I was looking at it. It seems to require a good deal of additional
> overhead and complexity for just a “hey, where are you?” install check.
> Which will increase the memory resident footprint a little unnecessarily.
As per my first examples, the numbers are actually fairly close for a
minimal implementation of AMIS vs your proposal so far. To reiterate,
you don't have to provide all the possible AMIS interfaces, even a
reduced and possibly specification-noncompliant set of features is
better to provide on the AMIS interface than brewing your own. Even
minimal support for AMIS buys you more than your own custom interface.
> However, it is an (at least partially) accepted “standard” and possibly
> the better solution and would not increase the footprint that much. It
> is definitely worth further consideration.
Agreed.
>
>>
>> INT 2F - Multiplex - NOTES
>> AH = identifier of program which is to handle the interrupt
>> 00h-3Fh reserved for IBM (for DOS)
>> 40h-7Fh reserved for Microsoft (for DOS)
>> 80h-B7h reserved for IBM
>> B8h-BFh reserved for networks
>> C0h-FFh reserved for applications
>> AL is the function code
>> This is a general mechanism for verifying the presence of a TSR and
>> communicating with it.
>>
>> AMIS got introduced (by RBIL, sort of?) because so many apps use INT 2F,
>> but if you only need an install check, performance is no issue and you
>> can use INT 2F without problems.
>>
>> You can also use INT 2F just for detection and then acquire a pointer
>> to a fast call provided by your driver if you need something quick.
>
> I don’t think 2F would be a good choice. In part, you have the network
> redirector and other filesystem related functions on that interrupt.
> Every hook that gets installed there takes a little slice of time to
> process. Which in turn, can slow down normal file operations.
>
> Also, I think that there is to high a risk of having a possible
> collision with 2F with programs that just arbitrarily grab a fixed
> “unused” function to intercept.
Agreed on both counts.
> I think it comes down to either using 2D or 2B. I like 2B because there
> is almost no overhead and very simple to implement on both ends. But, 2D
> might be the better way to go and is a pre-existing semi-accepted
standard.
>
> :-)
I would argue that AMIS is actually better designed, and there is no
need to split up things with Yet Another Multiplex Interrupt standard.
On at 2023-04-21 11:04 -0400, jer...@shidel.net wrote:
>
>
>> On Apr 21, 2023, at 10:21 AM, tom ehlert <t...@drivesnapshot.de> wrote:
>>
>> Hi,
>>
>>
>>> At present, the interface program for Logger just performs a
slightly optimized brute force search for the Logger device driver.
Although reliable, it is very slow compared to providing a simple
interrupt call to test for installation.
>>
>> given that this detection will be done once per program start: how
many microseconds do you expect to save?
>
> Well, there is the catch.
>
> At present, the current semi-optimized brute force search is quick.
The delay in finding the driver is not really noticeable when a user
invokes the interface program to view, print or perform some other
operation.
>
> But there is a delay and those delays can add up. For example, if the
FreeDOS installer ran the interface program many times to put messages
into the log or write the results of programs like FDISK there for
debugging installation issues. All those executions will add up.
>
> But, like you said, how much performance will be gained? For example,
if adding a comment to the log takes 1 second to load the interface
program from floppy diskette and only 1/1000 of a second to find the
driver, then improving that makes almost no sense at all.
>
> I should probably benchmark it to see if it is worth the bother. But,
it would vary widely based on system and numerous other factors.
The AMIS route insures that you need to check at most 256 signatures, as
that is its limit of unsigned 8-bit multiplex numbers. But in the common
case, most of those multiplex numbers' installation check calls will
return al != 0FFh so you will usually scan much fewer signatures than
that. It is certainly going to be faster, and also more reliable, than
your current check.
>>> Looking at the different interrupts, I think I have come up with a
solution that will work well for Logger and any other driver or program
that needs such a check. So, I’d like to propose a “standard” we could
use. I’d like to get your feedback on what I’m thinking…
>>
>> setting a "standard" which is used probably exactly once ? we write
year 2023 ;)
>>
>> https://xkcd.com/927/
>>
>> INT 2D has been mentioned by others
>
> Maybe go for 16 instead. It is an even binary number. LOL
>
> XKCD is great.
I would also suggest, if AMIS doesn't do what you want it to do, you
have plenty of ways to strip off parts or add functions. The latter
either as functions specific to your multiplexer on the interrupt 2Dh
interface with private function numbers in al >= 10h, or on the private
callback returned by AMIS function 01h.
Or, you could propose extensions to the specification with new return
codes and/or new common AMIS function codes (al < 10h).
>>> Looking at RBIL, interrupt 0x2b is barely used by anything. Under
MS-DOS and FreeDOS, this simply points to an IRET. Under IBM ROM-DOS, AH
functions 0x00-0x03 do some things. But, all other calls do nothing.
>>
>>> https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html
<https://fd.lod.bz/rbil/interrup/dos_kernel/2b.html>
>>
>>> An install check issuing this interrupt would be simple to perform.
A program could set the Carry Flag, load AH/AX with “check for install”
function and set DS:BX to point to an identifier string (minimum 8
characters, no maximum). Then call the interrupt. On return, if the
Carry Flag is still set, then the install check failed. If the Carry
Flag is clear, it succeeded and other register would be modified to
according to the needs of the programs.
>>
>>> Implementation for Logger (as an example) could be:
>>
>>>
>>> CHECK:
>>> push cs
>>> pop ds ; set DS to our code segment
>>> stc ; set the carry flag
>>> mov ax, 0xfd00 ; set AX to install check
function
>>> mov bx, LOGGER_ID ; offset to LOGGER device
driver ID
>>> int 0x2b ; call install check interrupt
>>> jc NOT_INSTALLED ; nothing changed, driver
is not loaded
>>
>> ;you should add
>> cmp ax, MAGIC_VALUE
>> jne NOT_INSTALLED
>>
>> your proposed INT2B might be used by some other software that for
some crazy reasons
>> clears the carry flag.
>
> Your correct of course. :-)
>
> In implementation, I would probably have it return the pointer to the
driver header and verify the signature of the driver again by the caller.
Then it buys precisely nothing over using an AMIS-compliant installation
check.
>> anyway, INT2D is probably the better choice anyway.
>
> Probably, I just don’t like to add extra code and increase the
resident footprint without good cause.
>
> :-)
More on that later.
On at 2023-04-21 11:21 -0400, jer...@shidel.net wrote:
> Hi Eric,
>
>> On Apr 21, 2023, at 10:27 AM, Eric Auer <e.a...@jpberlin.de> wrote:
>>
>>
>> Hi Jerome,
>>
>> if you are worried about collisions, use AMIS int 2d.
>> Overhead is low and it is less crowded than MUX int 2f.
>>
>> RBIL was famous enough to make AMIS more widespread. FreeDOS is too
>> niche to re-invent that wheel and declare int 2b to be a new trend.
I agree with Eric.
>> Why would your RESIDENT driver have to do install checks?
>> You can do those in the TRANSIENT part and just store the
>> results as static data in the resident driver part.
>
> There driver does do an install check at start up to prevent multiple
instances.
> However, all of that code is discarded after initialization. But, you
still need to
> “store the results” in the resident portion and respond to the
interrupt calls
> properly.
>
> There is probably little difference in the resident portion between
using 2D or
> 2B. Only, writing both versions could say for certain which is larger
and by
> how much. But, RBIL says 64-bytes + 22 per hooked interrupt. A simple ID
> string comparison and set return values won't use much.
>
> But we are taking about bytes and the savings may not be worth it.
The numbers from the interrupt list are about a minimum fully compliant
multiplexer. You don't have to follow all of that.
Regardless, the "22 bytes per hooked interrupt" is mainly due to needing
a 18-byte structure known as the IBM Interrupt Sharing Protocol (IISP)
header for each hooked interrupt. Besides AMIS, I also want to advertise
the use of the IISP. You can use it regardless of whether your
multiplexer uses AMIS or not. And there's two sides to using it, you can
benefit from other programs providing these headers to you, as well as
you can offer them from your resident program.
If you want to learn what it is and what it is for, I can offer my
KEEPHOOK manual [7] as well as Chris Dunford's document on the protocol,
INTSHARE.DOC (an ASCII plain text file), which I have mirrored on our
server [8].
[7]: https://pushbx.org/ecm/doc/tsr/keephook.htm
[8]: https://pushbx.org/ecm/test/20211110/intshare/INTSHARE.DOC
>> So the risk of wasting RAM by using AMIS are minimal :-)
>>
>> I agree that MUX can get crowded. However, for disk I/O,
>> the actual I/O still is the main bottleneck (unless you
>> are a large cache with a very inefficient algorithm) and
>> you would not do install checks for each invocation anyway.
>>
>> One example of slow call chains which you can feel may
>> be old ANSI variants and slow system and VGA BIOS. But
>> that was long ago and even those were not THAT slow.
>>
>> AMIS can be expected to be much faster than MUX and it
>> will not be in the way for your CDEX performance either.
>>
>> If you want to avoid the hassle of scanning for a free
>> magic number, I would just hardcode one. Collision risk
>> will still be lower than when grabbing the whole int 2b.
I disagree, one of AMIS's greatest strength is the flexibility and
common interface to use. Scanning for a free multiplex number is a
transient operation for most any AMIS multiplexer. So I would not
recommend "hardcoding" an AMIS multiplex number.
>> I agree with Tom: Use int 2d, it is the better choice and
>> 2023 is not the year for MORE standards for simple things.
>
> The only advantages I see in using 2B instead are as follows, it is
> very light weight, it uses very little memory to implement, it has
> almost no chance of conflicts and only being used for an install check
> there is no impact on performance of any aspect of the system.
>
> Unless I just stay with the brute force search, 2D is probably the
> better choice. There are even suggestions that memory managers
> can query it to retrieve the driver name. Although, I don’t know
> if the FreeDOS mem command makes use of that or not.
MEM does not, but tools like our AMITSRS [9] can. My debugger also can
scan the AMIS interrupt lists in its "DIL" command [10]. And of course
the advanced deinstallation method described in the KEEPHOOK manual can
benefit from using AMIS multiplexers and/or IISP headers.
Something not yet mentioned is that you can make use of AMIS, to some
degree, to implement uninstallation of your multiplexer as well. Some
notes on that:
* If you return codes 03h or 04h or 06h or 07h from the AMIS uninstall
function, you should have already deallocated resources like your XMS
memory and additional DOS memory blocks. If you do not do that, then the
REMOVE tool may attempt to uninstall you by unhooking all interrupts,
unlinking device headers, and freeing your main memory block. This is an
area of AMIS that could still see improvements.
* You can again make use of other programs' AMIS and IISP use.
* It is possible to uninstall a device driver installed from
(FD)CONFIG.SYS, for an example refer to the description of the QC
command in lDebug's manual section on "Invoking the debugger as a device
driver" [11] or the source that implements the QC command [12] [13].
Furthermore, I would also suggest to merge the two executables into one
dual-mode program. You can do either a .COM style flat-format executable
that doubles as a flat-format device driver, like so in DRVEXCH [14]:
section .text
org 0 ; it is a device driver
; device driver header
devdrvrhdr
jmp short runcom ; will be patched to -1 later
dw -1 ; pointer to next device driver header
dw 1000000000000000b ; device driver attributes (character device
driver)
dw stratfunc ; offset to device driver strategy routine
dw intfunc ; offset to device driver interrupt routine
db "DRVEXCH$" ; device driver name
... Or, you can make an MZ .EXE style segmented executable that also can
be loaded as a device driver (like the original Microsoft EMM386.EXE).
You can implement this in the same way as the dual-mode flat-format file
(a jump in the device header that is overwritten with -1 by the device
init function), and/or just set up a different entrypoint for
application mode in your MZ header.
[9]: https://pushbx.org/ecm/web/#projects-amitsrs
[10]: https://pushbx.org/ecm/doc/ldebug.htm#cmddi
[11]: https://pushbx.org/ecm/doc/ldebug.htm#invoking-device
[12]:
https://hg.pushbx.org/ecm/ldebug/file/ae5ecb16ae6d/source/debug.asm#l4580
[13]:
https://hg.pushbx.org/ecm/ldebug/file/ae5ecb16ae6d/source/debug.asm#l5175
[14]: https://www.bttr-software.de/products/drvexch/
Regards,
ecm
_______________________________________________
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel
