Re: [Freedos-devel] Analysing KITTENC

[perditionc]

On Fri, Aug 20, 2021, 6:46 PM Aitor Santamaría  wrote:

> Hello Jeremy,
>
> ...
>
> Thus sprintf and processEscChars differ in behaviour (although for this
> "buggy" string definition). I suppose that this new function
> (processEscChars) is added to avoid problems with the % symbol...
>
> Aitor
>

Yes, it was originally added so I could include newlines, the others added
to be compatible with c strings and sprintf avoided exactly as you
suggested to avoid issues with %.

Jeremy
___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Analysing KITTENC

[Aitor Santamaría]

Hello Jeremy,

In this case (strtoul), the code always assumes the \xdd has only two
numeric characters, i.e. a byte. I suppose that makes sense for DOS
compilers (although the limitation doesn't seem to exist in general:  Escape
sequences in C - Wikipedia
 ). Thus "\x208"
translates to " 8".
For me \x208 is a bug in the string design for the DOS environment, not "
8".
Curiously, OpenWatcom C  takes only the LAST two digits of the number to
make the character. Thus \x208 becomes \x08 (backspace).

Thus sprintf and processEscChars differ in behaviour (although for this
"buggy" string definition). I suppose that this new function
(processEscChars) is added to avoid problems with the % symbol...

Aitor


On Sat, 21 Aug 2021 at 00:04,  wrote:

> On Fri, Aug 20, 2021 at 4:48 PM Aitor Santamaría 
> wrote:
> >
> > Hi,
> >
> > Thanks Jeremy.
> >
> > On Fri, 20 Aug 2021 at 13:03,  wrote:
> >>
> >> > I was trying to figure out how KITTEN/KITTENC works. Seems to attach
> each NLS file with the "KITTENC" signal by the end of the executable. I've
> read about making UPX support easier, I guess you UPX before you ATTACH.
> >>
> >> It is written to support either, appending before or after UPX, it
> >> stores the original file size and then when ran uses the compressed
> >> file size computing the difference and adjusting the stored offsets
> >> with this difference.
> >> So if you don't compress or append the languages files after
> >> compressing the difference is 0, but if you do it before compressing
> >> then the difference is your compression savings more or less and
> >> offsets adjusted accordingly.
> >
> >
> > But if you append after compressing, wouldn't UPX mess the attached
> stuff?  (I mean if it tries to "expand" something that hasn't been
> compressed).
> > I suppose it is tested to work, but I am curious, I am not very familiar
> with the technical details of how UPX works.
> >
>
> I haven't looked into what exactly it does, but it matters whether you
> are compressing an exe or a com file.
> For exe files, it uses the header (I believe) and only compresses the
> actual executable portion, not anything appended to the file, but
> instead anything after the end of executable portion is copied asis to
> the end of the compressed portion.
> For com files, there is no header, so it can't determine the end of
> the executable portion and anything appended will be treated as part
> of the executable program and thus compressed.  So kittenc and similar
> tools will not work if you append to a com file and then upx - though
> I suppose they could be modified in that case to search memory for the
> strings as upx should both compress and decompress the appended data.
> If you append after compressing, I'm not sure exactly what the upx
> stub does, whether the uncompression has a way to tell the end of the
> compressed data or it will try to uncompress data that wasn't
> compressed (I did a quick but not exhaustive check and a upx com file,
> appending junk, running the com file does not produce any errors
> suggesting upx stub has a way to stop at the end of compressed data).
>  [Note: upx limits com files to about 65350 bytes per its
> documentation]
>
>
> >> > Why did you need to write an own function to convert string to UL?
> >>
> >> It was from the original kitten - not needed in kittenc since that
> >> part occurs during build, but also no need to change working code;  I
> >> like the update from switch statement to strchr for the xlat portion
> >> of the escape conversion
> >
> > Well I see it used when processing the Esc chars, when a language file
> is processed.
> > What I was wondering is if the function is smaller, or has some benefits
> over using the standard strtoul.
> >
> > Aitor
>
> I don't recall (pretty sure it wasn't me who added it), but my guess
> it was either for smaller size like you suggest (it definitely can be
> smaller) and/or because some compilers lack strtoul (not sure how
> many, but at least one lacks it).  However, it should be noted that
> mystrtoul does not match strtoul:
> int mystrtoul (char *src, int base, int size);
> vs
> unsigned long int strtoul (const char* str, char** endptr, int base);
> where src and base are the same for both, one returns a long the other
> an int,  mystrtoul uses size to determine how many numerical
> characters to process whereas strtoul updates endptr to point next
> char after number (if not NULL)
>
>
> Jeremy
>
>
> ___
> Freedos-devel mailing list
> Freedos-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-devel
>
___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Analysing KITTENC

[perditionc]

On Fri, Aug 20, 2021 at 4:48 PM Aitor Santamaría  wrote:
>
> Hi,
>
> Thanks Jeremy.
>
> On Fri, 20 Aug 2021 at 13:03,  wrote:
>>
>> > I was trying to figure out how KITTEN/KITTENC works. Seems to attach each 
>> > NLS file with the "KITTENC" signal by the end of the executable. I've read 
>> > about making UPX support easier, I guess you UPX before you ATTACH.
>>
>> It is written to support either, appending before or after UPX, it
>> stores the original file size and then when ran uses the compressed
>> file size computing the difference and adjusting the stored offsets
>> with this difference.
>> So if you don't compress or append the languages files after
>> compressing the difference is 0, but if you do it before compressing
>> then the difference is your compression savings more or less and
>> offsets adjusted accordingly.
>
>
> But if you append after compressing, wouldn't UPX mess the attached stuff?  
> (I mean if it tries to "expand" something that hasn't been compressed).
> I suppose it is tested to work, but I am curious, I am not very familiar with 
> the technical details of how UPX works.
>

I haven't looked into what exactly it does, but it matters whether you
are compressing an exe or a com file.
For exe files, it uses the header (I believe) and only compresses the
actual executable portion, not anything appended to the file, but
instead anything after the end of executable portion is copied asis to
the end of the compressed portion.
For com files, there is no header, so it can't determine the end of
the executable portion and anything appended will be treated as part
of the executable program and thus compressed.  So kittenc and similar
tools will not work if you append to a com file and then upx - though
I suppose they could be modified in that case to search memory for the
strings as upx should both compress and decompress the appended data.
If you append after compressing, I'm not sure exactly what the upx
stub does, whether the uncompression has a way to tell the end of the
compressed data or it will try to uncompress data that wasn't
compressed (I did a quick but not exhaustive check and a upx com file,
appending junk, running the com file does not produce any errors
suggesting upx stub has a way to stop at the end of compressed data).
 [Note: upx limits com files to about 65350 bytes per its
documentation]


>> > Why did you need to write an own function to convert string to UL?
>>
>> It was from the original kitten - not needed in kittenc since that
>> part occurs during build, but also no need to change working code;  I
>> like the update from switch statement to strchr for the xlat portion
>> of the escape conversion
>
> Well I see it used when processing the Esc chars, when a language file is 
> processed.
> What I was wondering is if the function is smaller, or has some benefits over 
> using the standard strtoul.
>
> Aitor

I don't recall (pretty sure it wasn't me who added it), but my guess
it was either for smaller size like you suggest (it definitely can be
smaller) and/or because some compilers lack strtoul (not sure how
many, but at least one lacks it).  However, it should be noted that
mystrtoul does not match strtoul:
int mystrtoul (char *src, int base, int size);
vs
unsigned long int strtoul (const char* str, char** endptr, int base);
where src and base are the same for both, one returns a long the other
an int,  mystrtoul uses size to determine how many numerical
characters to process whereas strtoul updates endptr to point next
char after number (if not NULL)


Jeremy


___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Analysing KITTENC

[Aitor Santamaría]

Hi,

Thanks Jeremy.

On Fri, 20 Aug 2021 at 13:03,  wrote:

> > I was trying to figure out how KITTEN/KITTENC works. Seems to attach
> each NLS file with the "KITTENC" signal by the end of the executable. I've
> read about making UPX support easier, I guess you UPX before you ATTACH.
>
> It is written to support either, appending before or after UPX, it
> stores the original file size and then when ran uses the compressed
> file size computing the difference and adjusting the stored offsets
> with this difference.
> So if you don't compress or append the languages files after
> compressing the difference is 0, but if you do it before compressing
> then the difference is your compression savings more or less and
> offsets adjusted accordingly.
>

But if you append after compressing, wouldn't UPX mess the attached stuff?
(I mean if it tries to "expand" something that hasn't been compressed).
I suppose it is tested to work, but I am curious, I am not very familiar
with the technical details of how UPX works.

> Why did you need to write an own function to convert string to UL?
>
> It was from the original kitten - not needed in kittenc since that
> part occurs during build, but also no need to change working code;  I
> like the update from switch statement to strchr for the xlat portion
> of the escape conversion
>
Well I see it used when processing the Esc chars, when a language file is
processed.
What I was wondering is if the function is smaller, or has some benefits
over using the standard strtoul.

Aitor
___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Analysing KITTENC

[perditionc]

This is my understanding from reading the code a few days ago:

On Fri, Aug 20, 2021 at 6:32 AM Aitor Santamaría  wrote:
>
> Hello,
>
> Sorry about the SPAM. I found the older thread too long, so I'm starting a 
> new one with the same content (please comment here if you want to ).
>
> I was trying to figure out how KITTEN/KITTENC works. Seems to attach each NLS 
> file with the "KITTENC" signal by the end of the executable. I've read about 
> making UPX support easier, I guess you UPX before you ATTACH.

It is written to support either, appending before or after UPX, it
stores the original file size and then when ran uses the compressed
file size computing the difference and adjusting the stored offsets
with this difference.
So if you don't compress or append the languages files after
compressing the difference is 0, but if you do it before compressing
then the difference is your compression savings more or less and
offsets adjusted accordingly.

>
> Why did you need to write an own function to convert string to UL?

It was from the original kitten - not needed in kittenc since that
part occurs during build, but also no need to change working code;  I
like the update from switch statement to strchr for the xlat portion
of the escape conversion

>
> By the way, apparently I've found a typo:
>
> if (language &&
> _toupper(language[0]) == 'E' &&
> _toupper(language[0]) == 'N')
> {
> // set LANG=EN switches to internal strings
>

An interesting space savings, assuming hard coded strings are English
and that the English catalog has the same text - which is probably
true 100% of the time.

>
> Best wishes,
> Aitor

Jeremy


___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

[Freedos-devel] Analysing KITTENC

[Aitor Santamaría]

Hello,

Sorry about the SPAM. I found the older thread too long, so I'm starting a
new one with the same content (please comment here if you want to ).

I was trying to figure out how KITTEN/KITTENC works. Seems to attach each
NLS file with the "KITTENC" signal by the end of the executable. I've read
about making UPX support easier, I guess you UPX before you ATTACH.

Why did you need to write an own function to convert string to UL?

By the way, apparently I've found a typo:

if (language &&
_toupper(language[0]) == 'E' &&
_toupper(language[0]) == 'N')
{
// set LANG=EN switches to internal strings


Best wishes,
Aitor
___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: [Freedos-devel] Devel-Philosophy [was: blocking bugs/issues for FreeDOS 1.3]

[Aitor Santamaría]

Hi,

I was trying to figure out how KITTEN/KITTENC works. Seems to attach each
NLS file with the "KITTENC" signal by the end of the executable. I've read
about making UPX support easier, I guess you UPX before you ATTACH.

Why did you need to write an own function to convert string to UL?

By the way, apparently I've found a typo:

if (language &&
_toupper(language[0]) == 'E' &&
_toupper(language[0]) == 'N')
{
// set LANG=EN switches to internal strings


Best wishes,
Aitor



On Fri, 6 Aug 2021 at 20:02, tom ehlert  wrote:

> Hi Robert,
>
> am Freitag, 6. August 2021 um 19:44 schrieben Sie:
>
> > Hi Jim,
>
> >> Good to see this update. I look forward to trying this out.
> >>
> >> One comment from reviewing the code - you use this comment at the top
> >> of both kittenc.c and kitten.c:
> >>
> >>> /*
> >>>   This software is free software; free to use,
> >>>   modify, pass to others, whatever
> >>>
> >>>   use it at your own risk
> >>> */
> >>>
> >>
> >> However, you don't include a license file (usually LICENSE.TXT or
> >> COPYING.TXT) to indicate what "free software" license you are using.
> >> Can you be more specific? Is this under the GNU GPL? GNU LGPL? MIT?
> >> BSD? Apache? Or some other license?
>
> > His comment sounds like Public Domain to me.
>
> it's intended to be as free as possible. I don't even care if the evil
> satan (MS or Oracle) uses, modifies or whatever does with it.
>
>
> > kitten.h is LGPL. Dunno if this is compatible.
> thanks for noting this. will change this as soon as possible.
>
> BTW: since the big fight of Oracle vs. Google wrt JAVA include files
> we all know that .H files aren't even copyrightable.
>
> also: by german law, for copyright claims your software (or other
> thing) has to have a minimum 'invention hight'. KITTEN.H certainly has
> no 'invention highness'.
>
> so: there shouldn't be a copyright on it in the first place.
>
>
>
> > I tend to dislike this license stuff...
> so do I
>
> Tom
>
>
>
> ___
> Freedos-devel mailing list
> Freedos-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-devel
>
___
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel