Re: Marshal and friends

[mratsim]

It doesn't use mmap AFAIK but for binary serialization NESM looks really nice: 
[https://xomachine.gitlab.io/NESM](https://xomachine.gitlab.io/NESM)/

Re: Marshal and friends

[cblake]

Well, fair point and a nice link. I mostly thought identical names for roughly 
identical things and worth mention.

Since this thread is about storing to disk, I should mention the too often 
neglected option of near-zero overhead native-binary-layout file formats that 
are "`mmap` & go" with no repeated parsing or even unnecessary paging. The file 
system can basically become your memory allocator. Ascii->binary and especially 
binary->Ascii is an expensive conversion for numbers (though any cost is always 
"compared to what").

Native binary does break "portability" to e.g. machines with different 
endianness/byte-order or non-IEEE floating point formats and whatnot. In the 
modern era, little endian and IEEE FP has won to such a degree that in practice 
this may matter little, if at all. In the old days, sharing a network volume 
between e.g., a big endian Sparc and a little endian Intel box was more common.

Anyway, Nim supports this all fine (as in, e.g., my 
[https://github.com/c-blake/suggest)](https://github.com/c-blake/suggest\)), 
but A) that is _far_ from pedagogical code, { but hey, at least it's a fully 
worked example }, and B) it can often be error prone work in general. Someone 
out in Nim land should maybe do a macro package someday to make it easier 
(unless they already have?).

Re: Marshal and friends

[sschwarzer]

I think, as far as usage is concerned, Nim's `marshal` module corresponds more 
to Python's `pickle` module. Compared to the `pickle` module, the `marshal` 
module has quite a few caveats: 
[https://docs.python.org/3/library/pickle.html#comparison-with-marshal](https://docs.python.org/3/library/pickle.html#comparison-with-marshal)

Re: Marshal and friends

[cblake]

Totally agreed, but it bears noting that Python also has a module named 
`marshal` with a `dump` and `load` that has generally been much faster/more 
space efficient than `pickle`/ `cPickle`. Basically "pickle but even less 
readable". So, one could also think of Nim `marshal` as like Python `marshal`. 
:-)

Re: Marshal and friends

[mratsim]

Marshaling is not about being human-readable though. The fact that it uses json 
is just because it was the easiest and it may change in the future.

Marshalling is just about dumping in-memory data so that it can be reloaded at 
a later time.

Think of it as Python pickle:

  * no guarantees of human readability
  * no guarantees of format stability

Re: Marshal and friends

[Vindaar]

> For JSON to() macro we have
> 
> > Heterogeneous arrays are not supported.
> 
> I have never seen that term in Nim world before ???

That just refers to the fact that in JSON you can of course have a 
heterogeneous array, like:


let heterogeneous = %* [1, 2.5, "Hello"]


Run

and these simply cannot be mapped to Nim types properly.

Re: Marshal and friends

[Stefan_Salewski]

After a closer look at JSON and YAML it seems that the user interface for 
storing and loading objects is similar, so I can start with one and maybe later 
exchange it with the other :-)

YAML is really large, a small test program generates a 200k executable in 
release mode.

For module marshal we have

> Restriction: For objects their type is not serialized. This means essentially 
> that it does not work if the object has some other runtime type than its 
> compiletime type.

Not a real problem for me, but does this restriction apply for JSON and YAML as 
well?

For JSON to() macro we have

> Heterogeneous arrays are not supported.

I have never seen that term in Nim world before ???

Re: Marshal and friends

[juancarlospaco]

JSON has a pretty proc that makes JSON pretty.

CI are dumb sometimes they fail for no reason or timeouts. 

Marshal and friends

[Stefan_Salewski]

[https://nim-lang.org/docs/marshal.html](https://nim-lang.org/docs/marshal.html)

I just started playing with marshal module. Seems to be the easiest way to 
store objects in an human readable form to disk. My test is


import
  marshal, streams

var s = newFileStream("somefile.txt", fmWrite)

type
  R = object
x, y, w, h: int
  
  O = object
name: string
r: seq[R]

var o: O
o.name = "test"

o.r.add(R(x: 1, y: 2, w: 2, h: 6))
o.r.add(R(x: 5, y: 7, w: 20, h: 8))
o.r.add(R(x: 9, y: 17, w: 28, h: 18))

store(s, o)
s.close()


Run

Result is


$ cat somefile.txt
{"name": "test", "r": [{"x": 1, "y": 2, "w": 2, "h": 6}, {"x": 5, "y": 7, 
"w": 20, "h": 8}, {"x": 9, "y": 17, "w": 28, "h": 18}]}


Run

So all is dumped to one single very long line? Then that is not really human 
readable. The other disadvantage of marshal is that it seems to work not with 
inheritance, it can only use compile time types, but that should be OK for my 
use case, as I try to avoid inheritance.

So I may have to use JSON or extern YAML module? I really like YAML, as it has 
good human readability and can be even edited with a text editor. And if I 
remember correctly its author was a bright one. But I think he has retired 
years ago, can not even remember his name.

So my question would be: What is the future of YAML module, will it at least 
still work with Nim 2.0 ?