Re: Printed representation of (char 0) ?

[Thorsten Jolitz]

Hi Alex and beneroth,
there are system interfaces that don't handle binary data and send them as
hex strings to middleware for conversion ...
All I wanted was a quick way to cross check that conversion with PicoLisp,
and that was really easy, except that NUL problem, solved now.
But very interesting discussion, I'm playing around with rd and pr right
now.
Thanks for the input
Cheers
Thorsten

Am Mi., 14. Feb. 2024 um 08:04 Uhr schrieb Alexander Burger <
picolisp@software-lab.de>:

> Hi Thorsten,
>
> > I wonder if there actually is a way to directly print ^@ in PicoLisp for
> a
> > "non-printable" hex "00", instead of NIL?
>
> As we see from the previous discussion, this is not an issue of
> printability.
> Other control characters may also be non-printable. It is an issue of
> binary data vs. symbol names.
>
> But you can of course print a caret and an at-mark instead of NIL
>
>(prin (or (something) "\^@"))
>
>
> > Wrt the application, I just have to deal with fixed length hex strings
> (!)
> > where the values at certain offsets carry semantics, conversions are
> done,
> > and it's crucial that values stay in that position, the NUL values
> matter.
>
> Yes, but why do you need to convert it to a string? I would process these
> data
> all exclusively as a list of numbers, and do the final printing explicitly
> (if
> needed at all). This printing may print '0' as "\^@", and also take care of
> other control characters and non-printable stuff.
>
> ☺/ A!ex
>
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>

Re: Printed representation of (char 0) ?

[Alexander Burger]

Hi Thorsten,

> I wonder if there actually is a way to directly print ^@ in PicoLisp for a
> "non-printable" hex "00", instead of NIL?

As we see from the previous discussion, this is not an issue of printability.
Other control characters may also be non-printable. It is an issue of
binary data vs. symbol names.

But you can of course print a caret and an at-mark instead of NIL

   (prin (or (something) "\^@"))


> Wrt the application, I just have to deal with fixed length hex strings (!)
> where the values at certain offsets carry semantics, conversions are done,
> and it's crucial that values stay in that position, the NUL values matter.

Yes, but why do you need to convert it to a string? I would process these data
all exclusively as a list of numbers, and do the final printing explicitly (if
needed at all). This printing may print '0' as "\^@", and also take care of
other control characters and non-printable stuff.

☺/ A!ex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[picolisp]

Hi Thorsten,

I agree with the others, that's not a string you are processing.
Strictly speaking, PicoLisp text functions and PicoLisp strings (and 
symbol names) must not contain NULL character.


How about processing this fixed-size values in binary, using (rd 'cnt) 
and (wr 'cnt) ?


Important: The alternative argument of (rd) makes it work very 
differently, the default behavior with 'sym argument is for PLIO (the 
picolisp binary format).
It's also usually a bad idea to mix binary and text processing when 
working with the same input channel, unless you really know about how 
picolisp text procressing works (the character buffering, with a 
character potentially being 1-4 bytes), so careful with that.


Greetings,
beneroth

On 13.02.24 20:30, Thorsten Jolitz wrote:

Hi Alex, Thomas,
thanks for your input, this is actually what I was looking for :
image.png
or even better:
image.png
I wonder if there actually is a way to directly print ^@ in PicoLisp 
for a "non-printable" hex "00", instead of NIL?
Wrt the application, I just have to deal with fixed length hex strings 
(!) where the values at certain offsets carry semantics, conversions 
are done, and it's crucial that values stay in that position, the NUL 
values matter. And I don't want to write a PicoLisp application for 
this, I just wanted an easy way to produce the expected conversion 
result in PicoLisp as a reference for comparison, and the above 
solution is fine for that.


Am Di., 13. Feb. 2024 um 09:09 Uhr schrieb Alexander Burger 
:


Hi Thorsten,

> But shouldn't hex 23232424 print to something like ##^N^N$$
instead of
> ##$$ ?

The problem is that you try to handle binary data as symbols. This
is not a good
idea. Binary data are numbers.

First of all, do you really have a hex message? Where does
it come from? Normally I would expect a list of numbers
as obtained with e.g.

   (make (do 96 (link (rd 1

If it is really a hexadecimal string, you can obtain the list
of numbers with

   : (make (for (L (chop "23232424") (cut 2 'L)) (link (hex @
   -> (35 35 0 0 36 36)

☺/ A!ex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Thorsten Jolitz]

Hi Alex, Thomas,
thanks for your input, this is actually what I was looking for :
[image: image.png]
or even better:
[image: image.png]
I wonder if there actually is a way to directly print ^@ in PicoLisp for a
"non-printable" hex "00", instead of NIL?
Wrt the application, I just have to deal with fixed length hex strings (!)
where the values at certain offsets carry semantics, conversions are done,
and it's crucial that values stay in that position, the NUL values matter.
And I don't want to write a PicoLisp application for this, I just wanted an
easy way to produce the expected conversion result in PicoLisp as a
reference for comparison, and the above solution is fine for that.

Am Di., 13. Feb. 2024 um 09:09 Uhr schrieb Alexander Burger <
picolisp@software-lab.de>:

> Hi Thorsten,
>
> > But shouldn't hex 23232424 print to something like ##^N^N$$ instead
> of
> > ##$$ ?
>
> The problem is that you try to handle binary data as symbols. This is not
> a good
> idea. Binary data are numbers.
>
> First of all, do you really have a hex message? Where does
> it come from? Normally I would expect a list of numbers
> as obtained with e.g.
>
>(make (do 96 (link (rd 1
>
> If it is really a hexadecimal string, you can obtain the list
> of numbers with
>
>: (make (for (L (chop "23232424") (cut 2 'L)) (link (hex @
>-> (35 35 0 0 36 36)
>
> ☺/ A!ex
>
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>

Re: Printed representation of (char 0) ?

[Alexander Burger]

Hi Thorsten,

> But shouldn't hex 23232424 print to something like ##^N^N$$ instead of
> ##$$ ?

The problem is that you try to handle binary data as symbols. This is not a good
idea. Binary data are numbers.

First of all, do you really have a hex message? Where does
it come from? Normally I would expect a list of numbers
as obtained with e.g.

   (make (do 96 (link (rd 1

If it is really a hexadecimal string, you can obtain the list
of numbers with

   : (make (for (L (chop "23232424") (cut 2 'L)) (link (hex @
   -> (35 35 0 0 36 36)

☺/ A!ex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Tomas Hlavaty]

On Tue 13 Feb 2024 at 08:35, Thorsten Jolitz  wrote:
> But shouldn't hex 23232424 print to something like ##^N^N$$
> instead of

no

^N could mean SO character with byte value 14
see the program ascii

a NUL character would be displayed as ^@

> So the printed ASCII string (as char) carries all the information from the
> hex string, and can be converted back to the exact same hex string?

no, you are using wrong tool for the job

as Alex said, a picolisp string cannot contain NUL byte

> At least in some special cases when it's needed?

arbitrary binary data is not a string
parse the binary data properly

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Tomas Hlavaty]

On Tue 13 Feb 2024 at 08:28, Thorsten Jolitz  wrote:
> But when I have a hexadecimal message string with fixed length, and the
> positions inside the string carry semantics? A certain value in a certain
> position has a meaning?

it is not a string, at least not in picolisp/C/unix sense

you need to properly parse the binary data into your own representation
and also write your own code to translate your own representation to
that binary data

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Thorsten Jolitz]

Hi Alex,
But shouldn't hex 23232424 print to something like ##^N^N$$ instead of
##$$ ?
So the printed ASCII string (as char) carries all the information from the
hex string, and can be converted back to the exact same hex string?
At least in some special cases when it's needed?

Alexander Burger  schrieb am Di., 13. Feb. 2024,
07:56:

> Hi Thorsten,
>
> > it's been some time .. ;-)
>
> Welcome back! :)
>
>
> > I'm playing around a bit with hex<->ascii conversion in PicoLisp, and I
> > have the problem that (char 0) = NIL
> >
> >  (hex "00")
> > -> 0
> > : (char (hex "00"))
> > -> NIL
>
> This is correct.
>
> 'char' converts a number to a (transient) symbol here.
>
> A symbol's name is a string, a null-terminated sequence of UTF-8
> characters. In
> case of 'char', this string has a single character and a terminating null
> byte.
> This is the same as in other languages like C.
>
> So the number 65 gives a symbol "A":
>
>: (char 65)
>-> "A"
>
> But what happens with 0?
>
> It gives an empty string, i.e. a null-byte
>
>: (char 0)
>-> NIL
>
> and an empty string in PicoLisp is NIL.
>
>: ""
>-> NIL
>
> ☺/ A!ex
>
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>

Re: Printed representation of (char 0) ?

[Thorsten Jolitz]

Ok I understand that.
But when I have a hexadecimal message string with fixed length, and the
positions inside the string carry semantics? A certain value in a certain
position has a meaning?

Say fixed length is 96 like in the example above, and 2323 ist two start
chars, and then 01 or 02 is a handshake.

And this message has to be translated to ASCII to be understood by a third
party, that sends ASCII answers back, that has to be translated to hex.

This does not work when a round-trip conversion of 232301 results in 23231.

Tomas Hlavaty  schrieb am Di., 13. Feb. 2024,
07:49:

> On Tue 13 Feb 2024 at 00:25, Thorsten Jolitz 
> wrote:
> > I would like to achieve a roundtrip like this:
>
> why?
>
> NUL is often a string sentinel value
> so trying to use it as a character value
> will lead to issues
> do not do that
>
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>

Re: Printed representation of (char 0) ?

[Alexander Burger]

Hi Thorsten,

> it's been some time .. ;-)

Welcome back! :)


> I'm playing around a bit with hex<->ascii conversion in PicoLisp, and I
> have the problem that (char 0) = NIL
> 
>  (hex "00")
> -> 0
> : (char (hex "00"))
> -> NIL

This is correct.

'char' converts a number to a (transient) symbol here.

A symbol's name is a string, a null-terminated sequence of UTF-8 characters. In
case of 'char', this string has a single character and a terminating null byte.
This is the same as in other languages like C.

So the number 65 gives a symbol "A":

   : (char 65)
   -> "A"

But what happens with 0?

It gives an empty string, i.e. a null-byte

   : (char 0)
   -> NIL

and an empty string in PicoLisp is NIL.

   : ""
   -> NIL

☺/ A!ex

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Tomas Hlavaty]

On Tue 13 Feb 2024 at 00:25, Thorsten Jolitz  wrote:
> I would like to achieve a roundtrip like this:

why?

NUL is often a string sentinel value
so trying to use it as a character value
will lead to issues
do not do that

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Thorsten Jolitz]

I would like to achieve a roundtrip like this:
: (hex (char (char (hex "23"]
-> "23"

but not like this:
: (hex (char (char (hex "01"]
-> "1"
: (hex (char (char (hex "00"]
-> "0"

It should be possible to do this, so that W = X in the end:
: (setq X
"23230200010001000F00323032342D30322D303931302D35342D303520202424")
: (length X)
-> 96
: (setq Y (chop X))
: (length Y)
-> 96
: (setq W (pack (mapcar '((A) (hex (char A))) (make (while Y (link (char
(hex (pack (cut 2 'Y)]
->
"232320001010F000323032342D30322D303931302D35342D303520202424"
: (length W)
-> 72

Am Mo., 12. Feb. 2024 um 23:56 Uhr schrieb Tomas Hlavaty <
picolisp@software-lab.de>:

> On Mon 12 Feb 2024 at 23:25, Thorsten Jolitz 
> wrote:
> > Shouldn't the (char 0) representation print to something
> > like ^N or so too, like (char 1), (char 2) etc?
>
> ^@
>
> what are you trying to achieve?
>
> why not use base64, for example?
>
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>

Re: Printed representation of (char 0) ?

[Tomas Hlavaty]

On Mon 12 Feb 2024 at 23:25, Thorsten Jolitz  wrote:
> Shouldn't the (char 0) representation print to something
> like ^N or so too, like (char 1), (char 2) etc?

^@

what are you trying to achieve?

why not use base64, for example?

-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Re: Printed representation of (char 0) ?

[Thorsten Jolitz]

Ah sorry, the moment I sent the question, I figured out the answer :
this is just a side effect of the final (pack ..).

When using (str ...) I get what I want:
:  (str (make (while Y (link (char (hex (pack (cut 2 'Y)]
-> "\"#\" \"#\" \"\^B\" NIL NIL NIL \"\^A\" NIL \"\^A\" NIL \"\^O\" NIL NIL
NIL NIL NIL NIL NIL \"2\" \"0\" \"2\" \"4\" \"-\" \"0\" \"2\" \"-\" \"0\"
\"9\" \"1\" \"0\" \"-\" \"5\" \"4\" \"-\" \"0\" \"5\" \" \" \" \" NIL NIL
NIL NIL NIL NIL NIL NIL \"$\" \"$\""

although somehow I still think it would be fine to have 'pack that prints
the NIL too, in this special case:
: (pack (make (while Y (link (char (hex (pack (cut 2 'Y)]
-> "##^B^A^A^O2024-02-0910-54-05  $$"

Am Mo., 12. Feb. 2024 um 23:25 Uhr schrieb Thorsten Jolitz <
tjol...@gmail.com>:

> Hello List,
> it's been some time .. ;-)
>
> I'm playing around a bit with hex<->ascii conversion in PicoLisp, and I
> have the problem that (char 0) = NIL
>
>  (hex "00")
> -> 0
> : (char (hex "00"))
> -> NIL
>
>  and disappears from the resulting ascii string.
>
> : (setq X
> "23230200010001000F00323032342D30322D303931302D35342D303520202424")
> : (setq Y (chop X))
> : (pack (make (while Y (link (char (hex (pack (cut 2 'Y)]
> -> "##^B^A^A^O2024-02-0910-54-05  $$"
>
> So how would it be possible to convert the ascii string back to the
> original hex string, when all the "00" are gone in the ascii
> representation? Shouldn't the (char 0) representation print to something
> like ^N or so too, like (char 1), (char 2) etc?
>
> When I try this online converter, the "00" are maintained as blanks:
> Hex to ASCII Text String Converter (rapidtables.com)
> 
> [image: image.png]
>
> and here is another representation that seems to enable the hex -> ascii
> -> hex conversion, getting exactly the same hex string in the end as in the
> beginning (not exactly the same conversion as above! just an example how it
> might look) :
> [image: image.png]
> Cheers
> Thorsten
>

Printed representation of (char 0) ?

[Thorsten Jolitz]

Hello List,
it's been some time .. ;-)

I'm playing around a bit with hex<->ascii conversion in PicoLisp, and I
have the problem that (char 0) = NIL

 (hex "00")
-> 0
: (char (hex "00"))
-> NIL

 and disappears from the resulting ascii string.

: (setq X
"23230200010001000F00323032342D30322D303931302D35342D303520202424")
: (setq Y (chop X))
: (pack (make (while Y (link (char (hex (pack (cut 2 'Y)]
-> "##^B^A^A^O2024-02-0910-54-05  $$"

So how would it be possible to convert the ascii string back to the
original hex string, when all the "00" are gone in the ascii
representation? Shouldn't the (char 0) representation print to something
like ^N or so too, like (char 1), (char 2) etc?

When I try this online converter, the "00" are maintained as blanks:
Hex to ASCII Text String Converter (rapidtables.com)

[image: image.png]

and here is another representation that seems to enable the hex -> ascii ->
hex conversion, getting exactly the same hex string in the end as in the
beginning (not exactly the same conversion as above! just an example how it
might look) :
[image: image.png]
Cheers
Thorsten