Re: Sort a table by value
A Table has no ordering, so you’d have to first copy it to some form that does, like a seq of (int, string) tuples, and then sort that (e.g. using a comparator proc that returns the result of comparing the two strings.)
Re: Why Seq search is faster than Table search
> My question is why linear search of seq is faster than constant search of > tables? Sorry to pile on four days later, but the right question would be "why is my benchmark code producing crazy results?" since it's much more likely there's something wrong with your quick benchmark than with Nim's fairly-mature standard library. I made the same mistake a few weeks ago when I posted here saying there seemed to be a bad performance bottleneck in the standard lib ... but it turned out that the real problem was that the build config I was using did not produce an optimized build at all. Oops.
Re: getFileSize async and sync
I've never used async file I/O APIs, but my understanding is that they just cover reading/writing the file itself. The size is metadata that comes from the filesystem's catalog / inodes / whatever, not the file itself, and is very cheap to get. I suppose you _could_ get the file size asynchronously by spawning the filesystem call onto a background thread, but that seems like overkill...
Re: threads:on + gc:orc = C(!) compiler errors in unit tests
Turns out it's pretty trivial to reproduce. I've filed
[https://github.com/nim-lang/Nim/issues/14865](https://github.com/nim-lang/Nim/issues/14865)
.
$ cat test.nim
import unittest
test "foo":
check 3 + 4 == 7
$ nim c -r --threads:on --gc:orc test.nim
Hint: used config file
'/Users/snej/.choosenim/toolchains/nim-#devel/config/nim.cfg' [Conf]
Hint: used config file
'/Users/snej/.choosenim/toolchains/nim-#devel/config/config.nims' [Conf]
/Users/snej/.choosenim/toolchains/nim-#devel/lib/pure/collections/setimpl.nim(43,
32) Hint: passing 'n[i].key' to a sink parameter introduces an implicit copy;
if possible, rearrange your program's control flow to prevent it [Performance]
CC: test.nim
/Users/snej/.cache/nim/test_d/@mtest.nim.c:888:88: error: incomplete
definition of type 'struct tySequence__shvncLWJZ4vpydQXcCi7CA_Content'
formatter__bVJh0BzzmxXcXxRmZHEkHg =
_->formatters__55U7g4zhjHV9bCI6emTgDiA.p->data[i];
~^
...
Run
threads:on + gc:orc = C(!) compiler errors in unit tests
OK, here's a weird bug. Once I enable both `--threads:on` and `--gc:orc`, all of my unit tests (which uses the `unittest` module) fail to compile, with gnarly looking C errors: /Users/snej/.cache/nim/testCodec_d/@mtestCodec.nim.c:1144:112: error: incomplete definition of type 'struct tySequence__shvncLWJZ4vpydQXcCi7CA_Content' ...eq___q9b1rGCpRgO9aMrUqDNKr8xg(_, NimTV_->formatters__55U7g4zhjHV9bCI6emTgDiA.p->data[i_3]); ~^ /Users/snej/.cache/nim/testCodec_d/@mtestCodec.nim.c:81:16: note: forward declaration of 'struct tySequence__shvncLWJZ4vpydQXcCi7CA_Content' typedef struct tySequence__shvncLWJZ4vpydQXcCi7CA_Content tySequence__shvncLWJZ4vpydQXcCi7CA_Content; Run This happens regardless of whether the test file in question uses any threading/async code. If I turn off either threads or ORC, all is well, except that the new code I'm writing definitely requires both of them. I took a look at the generated C file and it's definitely trying to use a forward-referenced but undeclared type. Beyond that I have no idea; the code's pretty dang well obfuscated! (Nim 1.2.2 on macOS 10.15.5. I'm using the Xcode 12 beta, but I don't believe the command-line tools are affected by that; `clang --version` still reports `Apple clang version 11.0.3 (clang-1103.0.32.62)`.)
Re: getFileSize async and sync
Not everyone uses async/await, and it's more common to use it for networking than for files (in my experience.) The async file I/O is for code that does use async and is worried enough about disk latency to want to avoid blocking on disk I/O. That's more extreme than I've ever gone, but I've never tried to build a high performance server...
Re: Multithreaded await
...OK, I have [some code that appears to work](https://gist.github.com/snej/653856212361be805751e4208a87d131), at least in Nim 1.2.2 with gc:orc. It lets you * Wrap a Future in another Future that can be passed to and completed on another thread, and will then invoke the original Future on its original thread (so it works correctly with `await`.) * `spawn` a proc and get its result as a `Future` instead of a `FlowVar`. Feedback is more than welcome.
Re: Multithreaded await
Yeah, I still have [your example](https://gist.github.com/treeform/3e8c3be53b2999d709dadc2bc2b4e097) open in a browser tab, and I think I may need to start borrowing from it, since it doesn't look like I can use Nim's thread pools with ORC (see above).
Re: Multithreaded await
Thanks for the example! I'm doing it slightly differently but using your basic technique. Next roadblock is that `FlowVar` doesn't seem compatible with ORC (which I need so I can pass object graphs across threads): `nim-1.2.2/lib/pure/concurrency/threadpool.nim(214, 6) Error: cannot bind another '=destroy' to: FlowVarObj; previous declaration was constructed here implicitly: nim-1.2.2/lib/pure/concurrency/threadpool.nim(211, 48)`
Multithreaded await
OK, I've got a WebSocket-based client running on a single thread with async/await and asyncdispatch. Now I want this to interact with other threads — for example, I'd like another thread to be able to post a message for the WebSocket to send. The sticking point is how to wake up the async dispatcher on its own thread. It seems like I need to have an async message queue, and a loop on my existing thread that pulls Futures from the channel and awaits them, much like the loop that reads from the WebSocket. I know how to build that ... but the Future is going to be completed on another thread, which means its callback will run on that other thread via `callSoon`, and it'll be handled by the wrong dispatcher. It looks like AsyncEvent is the tool to handle this ... ? I can create one, add it to my client thread's dispatcher, and then trigger it from arbitrary threads after I add something to my queue, and my callback will run on the client thread and can process the queue. I've seen the open issues with creating large numbers of AsyncEvents (they consume file descriptors) but that shouldn't be a problem for now because I believe I only need one. Even if I start doing async/await stuff on multiple threads, I should only need one per thread. Does this make sense? Or is there a better mechanism? (I'm not considering Weave, for now, because it looks like it's oriented toward finer-grained parallelism and is probably more complex than I need.)
Re: Benefit of the effect system?
What does the `{.raises:[...].}` pragma at the top of a source file do, as in
your ssz_serialization.nim? The manual only describes it as a proc annotation.
Re: New blog post: Ray tracing in Nim
Nice article! I like the explanation of Nim's benefits. I forwarded it to my team.
Re: Tables and Import
Module names seem to be lowercase by convention, and types uppercase.
Re: What's the future of "implicitDeref"?
> If I define the var version of the procedure, I don't need to define the ptr
> version anymore.
But the two have different domains (in the functional sense) — the `ptr`
parameter can take `nil`, while the `var` parameter can't. It seems wrong for
the compiler to produce an illegal behavior by implicitly dereferencing a
possibly-nil pointer and passing it to a function that will crash when it
accesses it.
In my experience, such crashes can be hard to debug. Both because your
reasoning can be wrong ("well, chooseAnim must have crashed because of this
other pointer variable, not because of `m`, because `m` isn't a pointer...")
and because you can sometimes pass this bogus nil var down into further levels
of the call stack before some unlucky proc gets around to dereferencing it, so
the crash can be pretty far away from the source of the bug.
The proc-vs-var issue you're bringing up is the same as "*" vs "&" in C++. Both
have valid purposes. You can either consistently use one or the other, or get
used to sprinkling "&" and "*" in your function calls … the latter is often a
code smell that you need to rethink the API.
Re: Nuglifier - Nim source code uglifier
EPiC It could support full 133tspEaK with a few tweaks to the lexer, just translating some digits to alpha chars in identifiers (3⟶E, 5⟶S, 4⟶A, etc.)
Re: Problem sending binary file by socket never ending.
> How I can send an EOF ? I tried "cL" or an empty string without success. You send an EOF by closing the socket. There's really no such thing as an "EOF character" ... it's just something made up by a library to return when the end of the file/data is reached. It doesn't exist in the file or over the socket. If you're trying to send a file over a socket but leave the socket open after the file is sent, then you need some other mechanism to tell the server when the file is done. The easiest way is to send the length (in bytes) of the file first, and then the data. The server reads the length, then reads that many bytes and writes them to the file, and then it knows the file is over. You _could_ make up some special byte like 0x00 to send at the end of the file ... but that byte could occur in the file itself, so then you need a way to escape it so the server doesn't think the file is over yet ... and then you need a way to escape the escape character. It's rather messy, and it slows down sending and receiving because you have to scan through all the bytes.
Re: Choosing Nim
There's also a probably unrelated [CodeRunner app](https://coderunnerapp.com) for macOS, that I'm pretty fond of. It's a scratch pad for writing and running programs in any of dozens of languages, everything from bash to Python to C++. It doesn't support Nim out of the box but I made a Nim plugin last month and announced it here; search the forum for "CodeRunner"...
Re: New garbage collector --gc:orc is a joy to use.
> You traverse the subgraph. In doing so you count the edges (= E) and sum the > RC fields (= S). A graph is isolated (sendable to a different thread) if and > only if S = E + 1. Clever! But what if there's an orphaned object cycle elsewhere that has a reference to an object in the subgraph? I've read that cycles are only cleaned up once in a while, so there's a time window where a dead cycle could still exist at the same time that I'm trying to make a cross-process call. I suppose when you detect a non-isolated subgraph, you could first force a cycle collection and then retry the isolation check, to see if it was a false positive. Is that the plan?
Re: Procedure overloading with explicit parameters
My understanding is that, in functional languages that use pattern matching, the different functions with the same name get merged together into one function by the compiler, with a switch/case expression to distinguish between the different cases. So Haskell's factorial :: (Integral a) => a -> a factorial 0 = 1 factorial n = n * factorial (n - 1) Run theoretically compiles into something very much like Nim's proc fac(n: int): int = if n == 0: 1 else: n * fac(n - 1) Run As others have already said, it's _runtime_ polymorphism, as opposed to the _compile-time_ polymorphism of Nim's (or C++s, etc.) function overloading. But you can do the same in Nim, it's just a syntactic difference.
Project-relative paths in nim.cfg
I need to give the compiler a path that's relative to my Nimble package. But it keeps interpreting it as relative to whatever directory the current source file is in. I added this to my `nim.cfg` file in the package root dir: --cincludes:"../../include/" Run When I run `nimble test`, the build fails because the include path is wrong: it's been constructed relative to the `tests` subdirectory, not the package root. I've seen the variable `$projectDir` used in other places so I thought that might fix it, but this doesn't work either: --cincludes:"$projectDir/../../include/" Run I can't find any real documentation of `nim.cfg` files. The compiler docs describe how they're located but doesn't explain their syntax or features. Some of the examples show all sorts of compiler-specific properties, but there's no description nor list of them. And then there's NimScript and `config.nims`, which is different (because it's a full Nim interpreter?) It's all pretty confusing.
Re: Mysterious compile error "system module needs: nimDestroyAndDispose" with --gc:orc
Seems to be a compiler bug. I've isolated it down to a tiny test case, and [filed an issue](https://github.com/nim-lang/Nim/issues/14813)
Re: Mysterious compile error "system module needs: nimDestroyAndDispose" with --gc:orc
With devel (downloaded/installed this morning) I get the exact same error: $ nimble test Executing task test in /Users/snej/Code/nim/blip/blip.nimble Verifying dependencies for blip@0.1.0 Info: Dependency on news@>= 0.5.0 already satisfied Verifying dependencies for news@0.5 Info: Dependency on zip@>= 0.2.1 already satisfied Verifying dependencies for zip@0.2.1 Compiling /Users/snej/Code/nim/blip/tests/testCodec (from package blip) using c backend /Users/snej/.choosenim/toolchains/nim-#devel/lib/system/assertions.nim(23, 11) template/generic instantiation of `sysFatal` from here /Users/snej/.choosenim/toolchains/nim-#devel/lib/system/fatal.nim(49, 5) Error: system module needs: nimDestroyAndDispose Tip: 5 messages have been suppressed, use --verbose to show them. Error: Execution failed with exit code 256 ... Command: "/Users/snej/.nimble/bin/nim" c --noNimblePath -d:NimblePkgVersion=0.1.0 --path:"/Users/snej/Code/nim/news/src" --path:"/Users/snej/Code/nim/zip" "-r" "--path:." "/Users/snej/Code/nim/blip/tests/testCodec" Run
Re: Copy-on-write container
I've read that C++ abandoned CoW for `std::string` because the atomic ref-counting turned out to be more expensive on average than simply copying the string every time. But of course YMMV; the tradeoff depends on the size of the objects and how expensive they are to copy. And for objects that won't be used concurrently on multiple threads, one can drop down to plain old ints for the refcounts (and skip the fences). That's probably the approach I'll use — I do need threads, but I'm going to try my best to enforce move semantics (a la Rust) for objects being sent between threads.
Re: Translate go function to nim
Is there something _specific_ in the code that you need help with?
Re: How to convert openarray[byte] to string?
I've filed [an issue](https://github.com/nim-lang/Nim/issues/14810) to request adding toString(openarray) to the standard library.
Re: How to convert openarray[byte] to string?
Interesting … I'm wondering where the 0 byte at the end of the cstring came from, since `cast` doesn't copy anything. You might just have gotten lucky in that particular example, and the memory after `bytes` happened to start with a zero. Here's a modification that fails, because the byte past the end of the openarray isn't zero: proc toString(bytes: openarray[byte]): cstring = assert bytes.len == 3 let str = cast[cstring](bytes) echo "length = ", str.len echo "str[0] = ", str[0].byte assert str.len == 3 return str var bytes = @[33'u8, 34, 35, 36, 37] echo toString(bytes.toOpenArray(0, 2)) Run
Re: Copy-on-write container
Did you watch Andreas's talk on ARC and ORC from Saturday's NimConf? He has a running example where he builds a toy `seq` from scratch, including the ref-counting. Basically you can implement your own ref-counting, if your object is a non-`ref`. You give it a `ptr` to the shared state, and put a refcount in the shared state, and then implement the ARC hooks like `=destroy` and `=sink`.
Re: New garbage collector --gc:orc is a joy to use.
Cool! I'm about to embark on multithreading, now that I've gotten my async networking code working on a single thread. Trying to switch over to gc:orc but [having a few problems](https://forum.nim-lang.org/t/6485). > Now you can just pass deeply nested ref objects between threads and it all > works. Is it really that simple? Because as @araq has stated, ARC's retain/release are _not_ atomic. That implies to me that a `ref` object can never be used concurrently on multiple threads. So I think by "pass" you mean "move" — the way you've described your code, it sounds like the work queues need to use move semantics, so the "push" operation takes an object as a `sink` parameter. Is that accurate?
Mysterious compile error "system module needs: nimDestroyAndDispose" with --gc:orc
My code is working well, and I'm trying out `--gc:orc` now. So I added that
flag to my `nim.cfg` and ran `nimble build`. OK, it compiles. But then `nimble
test` fails as soon as it compiles anything. In fact, if I run `nim c
tests/.nim`, where "" is anything, _whether or not such a file exists_
, I get this error:
/Users/snej/.choosenim/toolchains/nim-1.2.2/lib/system/assertions.nim(22,
11) template/generic instantiation of `sysFatal` from here
/Users/snej/.choosenim/toolchains/nim-1.2.2/lib/system/fatal.nim(49, 5)
Error: system module needs: nimDestroyAndDispose
Run
My other source directories don't have this problem. The only thing special
about `tests/` is that it has a `config.nims` file, containing the line
`switch("path", "$projectDir/../src")`. I believe this was created
automatically by `nimble init`.
What's going on here??
Re: How to convert openarray[byte] to string?
> @oswjk solution is correct. It would be nice to have this in the standard library, so that one doesn't have to unleash `unsafeAddr` just to do a simple conversion. > openarray[byte] are not nul-terminated unlike strings and that would cause > issues if you are interfacing with C code that expect nul-terminated cstring. I'm not. The problem is in pure Nim -- the cast returns a garbage `string` object, as shown in the above example. It appears to be misinterpreting the raw bytes in the `openarray` as if they were a `string` object, so e.g. the string's length is the first bytes of the array interpreted as a little-endian int. Again, I don't know the exact semantics of Nim's `cast[]`, so this might just be misuse of it. But it's dangerous that it works with one type (`seq`) but fails with a conceptually similar type.
Re: Help, I can't return an openarray
**Thank you!** That did the trick. 拾
Re: Help, I can't return an openarray
An array type has a fixed size. My type is variable-sized. (That's the whole reason openarray exists, right?)
Help, I can't return an openarray
I'm implementing a lightweight sequence-like type. I want it to have a `toOpenArray` proc, like `seq` and `string` do, so my type will work with any function that takes an `openarray`. But the compiler refuses to consider it: as soon as it sees the return type it shuts me down. proc toOpenArray(s: MySeq): openarray[byte] = ^ Error: invalid type: 'openArray[byte]' in this context: 'proc (): openArray[byte]' for proc Run I s'pose this is part of the "openarray only works as a parameter type" limitation. Is there any workaround; a pragma or something? Or are only "magic" procs allowed to do this?
Re: "subsequence" type (like Go's "slice")
> Er, you can slice with toOpenArray, no need for the inRange: var stuff. That isn't enough, because the proc alters the range (that's why `inRange` is a `var`.) It's basically consuming part of the input array, and updates the start of the range so that on return it points to only the unconsumed bytes. I could have made the API so it returns the number of bytes consumed, but that makes the caller do more bookkeeping. I suppose to some extent this is just me going against the grain of Nim's idioms...)
How to convert openarray[byte] to string?
There's no documentation I can find about how to convert an array of bytes to a string (where the bytes are UTF-8, of course.) I've looked through the manual, tutorial and several library modules. So far I've been using `cast[string](...)`. This appears to work fine for `seq[byte]`; at least I haven't found any problems resulting from it. But yesterday I ran into a crash doing the same with an `openarray[byte]` — this does _not_ work, it produces a string whose contents and length are garbage. proc toString(bytes: openarray[byte]): string = let str = cast[string](bytes)# <--- the call in question; how to do this properly? echo "length = ", str.len echo "str[0] = ", str[0].byte assert str.len == 3 # FAILS: actual value is 0x232221 return str let bytes = @[33'u8, 34, 35]# Before you suggest it, appending a 0 does not help :) echo toString(bytes) # Without the asserts above, this will spew garbage or crash Run So I'm guessing what I'm doing is not kosher, even though it seems to work on a `seq`. After all, `cast[]` is documented as being low-level and dangerous. What's the right way? (Ideally it would be efficient, i.e. not create an intermediate `seq[byte]`, as I'm doing this in some low-level network code.)
Re: "subsequence" type (like Go's "slice")
I'm writing code that manipulates portions of `seq``s, and I keep creating proc parameter lists that include ``input: openarray[byte]; inRange: var Slice[int]`, and then inside the proc I keep writing stuff like `input[inRange.a + i]` I have a large C++ codebase that uses a custom `slice` type that simply contains a pointer and a length, and I've found it extremely useful over the years; it really cleans up code that deals with byte ranges. (C++17's `string_view` is pretty much the same thing.) Of course it's very unsafe, but I'm used to that in C++. I'd like a safer version of it (more like Go's) in Nim. So I'm implementing one now; your toy sequence class from your ARC/ORC talk is proving very useful!
"subsequence" type (like Go's "slice")
As I get more fluent in Nim, the main tool I find myself missing is a lightweight reference to a range of bytes in memory — something that points to a portion of a `seq` without copying it. The closest equivalent I know of is Go's `slice[T]`, which is pretty ubiquitous — it's actually rare in Go to pass or return an array itself. I'm assuming this doesn't exist in Nim itself; I've spent some time scanning through the standard library docs, esp. the `system` and `sequtils` modules, and I think I'd have found it if it were there. But there might be a Nimble package. This could be implemented pretty simply, if non-optimally; something like type subseq*[T] = object parent: ref seq[T] range: Slice[Natural] Run The "non-optimal" part is the double-indirection getting to the bytes; it needs a `ref seq` to establish ownership so the GC will keep the heap block alive. One could add a direct pointer to the start of the range, in which case `range.a` isn't needed, so we get type subseq*[T] = object parent: ref seq[T] base: UncheckedArray[T] len: uint Run Both of these become unsafe if the seq is mutated; obviously if it's shortened enough to invalidate the end of the range, and more subtly the second design's `base` is invalidated if the seq has to grow its buffer. Not sure how to deal with that — it seems to require a Rust-like restriction that a sequence can't be mutated while any slices on it are extant, but I don't think Nim-as-we-know-it has ways to enforce that.
Re: Is --gc:arc completely independent from the older ownership model?
> for example, Swift's abysmal performance Whatever performance you're referring to, I doubt it's because of ARC vs GC. Swift's ref-counting implementation is based on Objective-C's. Objective-C has transitioned through manual ref-counting, GC, and now ARC; the GC was uncomfortably slow, but ARC is much faster, around the same performance as the manual ref-counting. (Objective-C has been ref-counted since sometime in the 1990s.) FWIW, I'm using homemade atomic ref-counting in the C++ codebase I now work on, and I've never noticed the retain/release functions being significantly 'hot' in performance tests. Maybe because they're testing large-scale stuff like database access and networking, not tiny artificial things that spend all their time creating objects ;-)
Re: Is --gc:arc completely independent from the older ownership model?
> the biggest impact is the fact that Nim's ARC does not use atomic reference > counting, instead a complete SCC will be moved between threads Just watch out that the moving doesn't become the critical path! Erlang does this, by copying the data between its 'process' heaps, but in the early days of Couchbase Dustin Sallings found that to be a major performance killer in CouchDB. (And part of the reason Couchbase abandoned CouchDB, or rather tore it into pieces and replaced half of them with C code.) I think you've mentioned Pony elsewhere, though — their approach sounds much better; transfer _ownership_ of a heap object between threads without actually having to copy the bytes around. That would be awesome to have in Nim.
Re: Has anyone wrapped zlib or reimplemented deflate?
> I highly recommend > [https://github.com/jangko/snappy](https://github.com/jangko/snappy) The protocol I'm implementing specifies Deflate. (Also, Snappy is really comparable Deflate; it's faster but provides lower compression. We have at least one big customer who is pretty obsessed with bandwidth and would probably not be satisfied with Snappy over the wire...)
Re: Has anyone wrapped zlib or reimplemented deflate?
nim-lang/zip is quite broken, at least on macOS — fails with `Z_VERSION_ERROR` in `initDeflate2`. Turns out this is because Zlib cleverly has the client pass in `sizeof(z_stream)`, for compatibility checking, and the Nim declaration of this struct is way too small. And _that 's_ because this source file uses `int32` for C `long`, which is wrong on LP64 systems like macOS. I'm in the middle of fixing this. The zlib tests pass now except for two that use `GZIP_STREAM` mode; I've never used that mode so I have no idea what the problem is.
Re: Has anyone wrapped zlib or reimplemented deflate?
Awesome, thanks! I see nim-lang/zip in the Nimble directory now ... but I swear it didn't show up yesterday when I searched for "zip", "zlib", "deflate", "gzip", ... 類
Has anyone wrapped zlib or reimplemented deflate?
To implement the network protocol I'm working on, I need to compress/decompress data using the [deflate](https://en.wikipedia.org/wiki/DEFLATE) algorithm. The existing C++ protocol implementation uses zlib for this. I don't see any zlib wrappers or reimplementations in nimble, but thought I'd ask here, to see if anyone has done this already but not published it separately as a package. Otherwise I'll start building my own zlib wrapper, and eventually publish it to nimble.
Re: Nim version 1.2.2 is out!
I just filed [https://github.com/nim-lang/Nim/issues/14719](https://github.com/nim-lang/Nim/issues/14719) to request this.
Re: Perf: Table.del(key)is taking 85% of my code's time
> Are you putting -d:release in the config file by any chance? If you do - that > wouldn't work, you must specify it in the call to the compiler/nimble build That's it! I put it in `nim.cfg` like all the other compile options. If instead I run `nimble build -d:release`, I get an optimized build. The help text for `nimble` doesn't describe the options to `build`, it just says `[opts, ...] [bin]`. And I already knew I could put `nim c` option flags in the config file, so it seemed obvious to put `-d:release` there. Help me understand the build process better -- why is it different putting it on the command line vs. in the config file?
Re: Perf: Table.del(key)is taking 85% of my code's time
> But specifying --opt:size at the same time may be not that good, do you use > -O3 and -Os for gcc at the same time? Most people do not. They're mutually exclusive, so the compiler's going to pick one if both are given. It looks as though the -Os wins since I do see a large reduction in code size. I'm of the belief that -Os is the best setting in most cases. (That was definitely the rule when I worked at Apple in the '00s.) Smaller code is more cache-friendly, launches faster, and contributes less to memory issues. If necessary you can use Clang or GCC function attributes to force individual functions to optimize for speed.
Re: Perf: Table.del(key)is taking 85% of my code's time
I looked at the Compiler User Guide, and it also implies that `-d:release` is sufficient to turn on compiler optimizations. As do your comments above. Weird... Here's my environment: * MacBook Pro, macOS 10.15.5 * Xcode 11.5; `clang --version` reports "Apple clang version 11.0.3 (clang-1103.0.32.62)" * I used to have Nim 1.2 installed from HomeBrew; this morning I uninstalled that, installed choosenim, and installed 1.2.2. I'm building a Nimble package that I created a few days ago, which imports asynchttpserver and the 'news' WebSocket library. I'm using `nimble build` to compile. My `nim.cfg` file contains: --path:src --outdir:bin --hints:off Run I'll use binary size as a rough indicator of optimization, for now. * Default (debug): binary size is 752K * With `-d:release`: 701K * With `-d:release --opt:speed`: 608K * With `-d:release --opt:size`: 528K * With `-d:release --opt:size --stackTrace:off --lineTrace:off --assertions:off --checks:off`: 273K * With `-d:danger`: 752K * With `-d:danger --opt:speed`: 647K * With `-d: danger --opt:size`: 570K * With `-d: danger --opt:size --stackTrace:off --lineTrace:off --assertions:off --checks:off`: 307K Conclusions: It definitely appears `-d:release` is not turning on optimizations. And it's weird that `-d:danger` results in a larger binary; it should be the other way around since it disables generating runtime checks, right?
Re: Perf: Table.del(key)is taking 85% of my code's time
Sorry, false alarm! I was misled by the `nim` command's help text: --opt:none|speed|size optimize not at all or for speed|size Note: use -d:release for a release build! Run **I took this to mean that ``-d:release`` was *sufficient* to get a fully optimized release build. Nope!** Once I started to look at the x86 disassembly it was painfully non-optimal. 冷 I changed the flags to `-d:release --opt:size --stackTrace:off --lineTrace:off --assertions:off --checks:off`, and everything got about 20 times faster... Now the code is spending almost all its time in `kevent` waiting for I/O, as it should be. **Question:** Is this the optimal set of flags for a release build? Is there an easy way to toggle between debug and release builds, without having to edit a bunch of lines in my `nim.cfg` file?
Perf: Table.del(key)is taking 85% of my code's time
I've got some networking code running now (implementation of a protocol based on WebSockets) and I have a little client and server that send each other messages as fast as they can. Now I get to profile it! (With `-d:release` of course.) Disappointingly, **about 85% of the total time is being spent in a single call to ``del(table,key)``** , where the Table's keys are `uint64` and the values are a `ref object` type. Here's a snippet what the macOS `sample` tool is showing me: 1614 pendingResponse__9bNzW9abL3s854sblP5selTw_2 (in client) + 434 [0x10d342b72] 1096 del__F9aUzT5FfSHbDaq0vvIxNng (in client) + 1935,1769,... [0x10d2e292f,0x10d2e2889,...] 518 del__F9aUzT5FfSHbDaq0vvIxNng (in client) + 1419 [0x10d2e272b] 277 isEmpty__5Qpmu5QqwDY0esVXn6wf7w_2tables (in client) + 48,23,... [0x10d2e2cc0,0x10d2e2ca7,...] 180 isEmpty__5Qpmu5QqwDY0esVXn6wf7w_2tables (in client) + 57 [0x10d2e2cc9] 180 nimFrame (in client) + 1,15,... [0x10d2e04d1,0x10d2e04df,...] 61 isEmpty__5Qpmu5QqwDY0esVXn6wf7w_2tables (in client) + 102 [0x10d2e2cf6] 61 popFrame (in client) + 1,14,... [0x10d2e0651,0x10d2e065e,...] Run _(Indentation shows the call nesting. The numbers on the left are the number of times the given stack frame was observed; there were 1886 samples total over 2 seconds.)_ I haven't dug into the Table source code yet, but it looks like there may be something really unoptimal there. The name-mangling is weird and very random-looking: are those base64 digests being appended to function names? **Is there a tool to unmangle names** so I can find out exactly what a specific function is? PS: From my past experiences with Go, I was expecting GC to be the culprit; there are a few stack frames that look like GC but they're wa down low, only a handful of samples. Nice! Is Nim using ARC by default now?
Re: std.sha1 secureHashFile memory usage
The name `secureHashFile` is sort of a misnomer, if the algorithm used is SHA-1...
Re: Nim's popularity
> I think Nim needs to ne on mobile. Since Nim runs on macOS I'm assuming it'll run on iOS — same toolchain, and 99% the same APIs until you get up to the GUI level. > There should be a simple way to write cross platform apps and especially > games in Nim. There's already a glut of cross-platform mobile app frameworks, in many different languages. And good GUI frameworks are really hard to write, especially if they need to work cross-platform. (Less so for games, since no one expects games to conform to a platform UI or support platform-specific features.)
Re: Why is my HTTP handler not gcsafe?
Weirdly, after I did some more work on my code I tried to reproduce the original problems I described (by backing out my tweaked versions of news and asynchttpserver) ... and they went away. Adding one single `gcsafe` pragma to the top-level HTTP server callback is sufficient. This is very strange, but I'm happy that the problem is gone.
Re: Why is my HTTP handler not gcsafe?
I ended up copying asynchttpserver.nim into my source tree, renaming it, and
deleting all the `{.gcsafe.}` pragmas. Everything now compiles, and it seems to
be working fine aside from the bugs in my own code that I'm ironing out.
¯_(ツ)_/¯
Re: Why is my HTTP handler not gcsafe?
> In short, no declaration of non-primitives global variable. There seems
> something declared at module level in blip that you didn't show.
My code has zero global variables.The global referred to in the error is
`nameIterVar`gensym21340263` which is obviously a compiler-generated name, not
anything I did myself.
I've also run into the error Warning: 'send' is not GC-safe as it performs an
indirect call here [GcUnsafe2] referring to this code:
type
MessageBuf* = object
# ...
sendProc: SendProc
SendProc* = proc(msg: sink MessageBuf): Future[MessageIn]
proc send*(buf: sink MessageBuf): Future[MessageIn] {.gcsafe.} =
return buf.sendProc(buf)
Run
So it appears that calling a closure isn't GC-safe?!
> Nim current approach force threads to talk explicitly.
I'm not using threads. This is all with asynchttpserver, which AFAIK is
single-threaded.
Re: Why is my HTTP handler not gcsafe?
I've added some more code, and now I'm getting actual errors that seem to
explain the cause of the GC-safe violation ... if I could understand them.
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(318, 24) Warning:
'receiveLoopNimAsyncContinue' is not GC-safe as it accesses
'nameIterVar`gensym21340263' which is a global using GC'ed memory [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/Users/snej/Code/nim/blip/src/blip.nim(126, 6) Warning: 'receiveLoop' is
not GC-safe as it calls 'receiveLoopNimAsyncContinue' [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/Users/snej/Code/nim/blip/src/blip.nim(136, 6) Warning: 'run' is not
GC-safe as it calls 'receiveLoop' [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(278, 31) Warning:
'cbIter' is not GC-safe as it calls 'run' [GcUnsafe2]
Run
`receiveLoop` is pretty simple (see below), but `receiveLoopNimAsyncContinue`
and `nameIterVar`gensym21340263` must be internal functions created by the
macros behind the `{.async.}` pragma, so I don't really have a clue what's
going on...
proc receiveLoop(blip: Blip) {.async.} =
while blip.socket.readyState == Open:
let packet = await blip.socket.receivePacket()
case packet.kind
of Binary: blip.handleFrame(cast[seq[byte]](packet.data))
of Close: return
else: continue
Run
Re: Why is my HTTP handler not gcsafe?
...also, at a higher level, why is gcsafe even an issue at all, when all this async stuff is supposed to be running on the same thread?!
Why is my HTTP handler not gcsafe?
I've now got some slightly less trivial async code I'm trying to get to run,
using the 'news' WebSockets module. The compiler complains that my HTTP handler
isn't gcsafe, and suggests "Annotate the proc with {.gcsafe.} to get extended
error information."
However,
1. I can't see how my code isn't GC-safe, since it has no global variables
whatsoever.
2. If I add the `gcsafe` pragma to my callback proc, it now compiles without
errors! Where's my "extended error information"?
import blip
import news, asyncdispatch, asynchttpserver
when isMainModule:
echo("Starting Blip server on port 9001...")
var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
if req.url.path == "/blipsync":
var ws = await newWebsocket(req)
echo "Creating new Blip"
var blip = newBlip(ws)
await blip.run() # <-- this line triggers the gcsafe error
echo "...Closed Blip"
await req.respond(Http404, "Nope", newHttpHeaders())
waitFor server.serve(Port(9001), cb)
Run
If I take out the line `await blip.run()`, the error goes away. But neither
that function nor anything it calls appears to involve any global state. (It's
too much code to post here in the forum, unless someone really wants me to!)
Did adding the `{.gcsafe.}` pragma silence a false alarm? Or is it suppressing
a genuine problem that will cause thread-safety issues?
Thanks!
—Jens
Re: Trivial AsyncHttpServer example crashes on every request
I confirmed that if I change line 5 to await req.respond(Http200, "Hello World", newHttpHeaders()) Run (passing in a non-nil headers object), the program works fine. I guess this has turned into a bug report, although when I started this post I didn't know what the problem was. Thanks for the rubber-duckie debugging, forum! What Github repo should I file this under? Nim itself?
Trivial AsyncHttpServer example crashes on every request
I'm exploring WebSockets and async/await. But I'm finding that the most trivial
example of an async HTTP server crashes on every request. If I run an exact
copy of the example from the asynchttpserver documentation:
import asynchttpserver, asyncdispatch
var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
await req.respond(Http200, "Hello World")
waitFor server.serve(Port(8080), cb)
Run
... it will crash hard when I send an HTTP GET to it, from any client I've
tried (Safari, curl, httpie):
/Users/snej/Code/nim/blip/src/server.nim(7) server
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(1886) waitFor
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(1576) poll
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(1340) runOnce
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(210)
processPendingCallbacks
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(37)
processRequestNimAsyncContinue
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asynchttpserver.nim(258)
processRequestIter
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(319) cb
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(34)
cbNimAsyncContinue
/Users/snej/Code/nim/blip/src/server.nim(5) cbIter
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asynchttpserver.nim(103) respond
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/httpcore.nim(180) hasKey
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Run
I don't know this code at all, but from quick inspection I can see that the
`respond` function in asynchttpserver.nim has a `headers` parameter that
defaults to `nil`, but it calls `headers.hasKey("Content-Length")` without
checking for `nil` first. Oops.
Is this module generally considered solid, and this is just an unusual bug
where no one ever tested the examples in the docs? Or is there another module I
should use instead?
—Jens
Re: Shared lib in Nim for other languages?
Go tends not to play well with other languages, because of its NIH philosophy — it has its own compiler, its own ABI, its own stack layout, its own heaps, its own threading. This makes it expensive to call in/out of Go.
Re: Dash docsets now available
Thanks for keeping this up to date! Is someone submitting a PR to Kapeli to update the Dash docset?
Re: Nim in business
> you might run into it lacking a package On the other hand, it's very easy to write wrappers for C APIs in Nim. Easier than any other language I've used except Swift.
Re: Can't pass a string to an openarray[char | byte] ?!?
Thanks for the info! But, why use a template? I tried your code in a playground, and it works just as well if I replace `template` with `proc`.
Can't pass a string to an openarray[char | byte] ?!?
If a function parameter is declared as `openarray[char]`, then I can pass a `string` to it. But if it's declared as `openarray[char | byte]`, then I can't. Huh? proc echoKey(key: openarray[char | byte]) = echo "Key: ", key var str = "hello" var bytes: seq[byte] = @[byte(1), 2, 3, 4] echoKey(bytes) # OK echoKey(str)# Error: type mismatch Run The full error message is: Error: type mismatch: got but expected one of: proc echoKey(key: openArray[char | byte]) first type mismatch at position: 1 required type for key: openArray[char or byte] but expression 'str' is of type: string Run So my next attempt is to explicitly add `string` to the union type: type Key = openarray[char | byte] | string proc echoKey(key: Key) = echo "Key: ", key echoKey(str)# OK echoKey(bytes) # Error: type mismatch: got Run Now strings work, but byte sequences don't! 勞 Error: type mismatch: got but expected one of: proc echoKey(key: Key) first type mismatch at position: 1 required type for key: Key but expression 'bytes' is of type: seq[byte] Run
Re: Lambda syntax is awkward
Oh! I tried using `block` and wondered why it didn't work ... I didn't realize parens were necessary. The `do` notation seems an improvement; it resolves two of the three complaints I gave above. If it had better type-inference I'd love it. Maybe that's why it's still experimental? e.g. what I'd like to see is `let plusTwo = twice(10) do (n):`
Lambda syntax is awkward
I'm finding Nim's syntax for calling lambdas awkward and verbose. proc twice[T](input: T, fn: proc(t: T):T):T = fn(fn(input)) let fourthPower = twice(10, proc(n: int): int = echo "Gratuitous echo!" return n * n ) Run * I have to declare parameter and return types. Even C++ lets me omit the return type! * The nested proc is still inside the outer proc's parentheses, so that has to be closed afterwards. * And a nitpick: the initial line ends with `=` not `:`, unlike all other statements that introduce nested blocks in a proc. **I 'm aware of the ``sugar`` module.** The `=>` macro makes this a lot better, but it only seems to work with single-line lambdas. let plusTwo = twice(10, (n) => n + 1) Run Every attempt I've made to use it in a multi-line lambda fails with syntax errors: let plusTwo = twice(10, (n) => echo "Another gratuitious echo" n + 1) # Error: expected: ')', but got: 'n' Run I like that templates and macros can be called with a block as a final parameter, but that feature is limited in that the block can't take any parameters and can't return a value. Is there something I've missed? Or are there proposals to improve this in the future? —Jens
Re: Creating a seq or openarray on unmanaged memory
> Only Rust has memory safe zero-copy collection that can be stored in a type > (their slice type). Not exactly. Only Rust has this _[plus](https://forum.nim-lang.org/postActivity.xml#plus) compile-time safety checks and zero runtime overhead. But Go's slices are memory-safe and zero-copy and can be stored, for example. It accomplishes this by having the slice type contain a hidden reference to the object that owns the memory; thus the GC ensures that a slice can't outlive its backing store. That would be pretty easy to support in Nim. I could easily make such an object myself, it just wouldn't work the same as an array/seq/string, as I said above. But thanks for the pointers [sic] to the "it" templates and zero-functional; I'll check those out.
Re: Best Nim translation of C {void*, size_t} struct
Just to make sure you're answering the exact question I asked :) — if I declare
a C proc that takes an `openarray`, when Nim calls that function does it pass
that parameter as a pointer followed by an int? For example:
proc set_bytes(bytes: openarray[byte]) {.importc: "set_bytes".}
let data: seq[byte] = @[1, 2, 3, 4]
set_bytes(foo, data)
Run
Will this work going forward? (I could try it out myself, but that only means
it works in Nim 1.2, not that it's expected/supported...)
Re: Nim Cheatsheet PDF (English+Spanish+Latex)
Nice! I made a copy that's black-and-white, better suited for laser-printing: [https://www.overleaf.com/read/gytjmgdzmcdt](https://www.overleaf.com/read/gytjmgdzmcdt)
Re: incorrect set-to-int conversion
> Note that enums with holes may become deprecated in future That would be silly IMHO — there are so many uses for enums that are non-consecutive. It seems like Nim's idea of enums is primarily "named ordered values you can make a set out of", and assigning specific numbers to them is considered a secondary use case. But that rules out too many of the real-world uses of enums, especially in bridging other APIs or implementing network protocols. > Araq sometimes recommends using distinct ints instead of enums That doesn't work with bit-sets (which are a great feature), but I might consider it for non-set-related enums like lists of error codes.
Re: Creating a seq or openarray on unmanaged memory
> Wrap the ptr UncheckedArray, len pair in an object I could do that, but this object would be a second-class citizen since it's neither an array nor seq nor string. I could implement `[]` and `len`, but it still wouldn't work with e.g. `sequtils`, right? I guess what I'm saying is that Nim doesn't seem to have a collection/sequence abstraction the way Swift, Rust, C++, Python etc. do.
incorrect set-to-int conversion
As part of bridging a set of bit-masks in a C API, I declared an enum and a set:
type
Flag = enum
A = 4
B = 6
# ... more flags...
Flags = set[Flag]
Run
After about half an hour of debugging an mysterious EINVAL return value from
the C API I finally discovered that **the set value have the wrong integer
values** :
check cast[int]({A}) == 16
check cast[int]({B}) == 64
Run
The checks fail — turns out `{A}` is 1 and `{B}` is 4! 勞
I figured out I can work around this by adding a fake enum item whose value is
0. So it appears that Nim's `set` always assigns bits relative to the lowest
enum value, instead of using the enum values as absolute bit positions.
Is this a bug, or intentional? If the latter, is there a cleaner workaround
like a pragma?
—Jens
PS: Don't ask me why this API doesn't use bits 0-3! I didn't write it. I
suspect there are some private flags, or maybe there used to be other flags but
they are obsolete.
Re: Creating a seq or openarray on unmanaged memory
I want to provide a (reasonably) safe interface, and returning an `UncheckedArray` clearly wouldn't be safe. The other approach I'm thinking of is to make the proc take a function parameter, and pass the `openarray` to the callback function. It makes the call site a bit ugly, but it'll be safe.
Creating a seq or openarray on unmanaged memory
I'm wrapping a C function, a getter that returns a `{const void*, size_t}`
struct pointing into memory managed by the C library. The obvious wrapper proc
would copy the bytes into a `seq[byte]` and return that.
However, this C library is a high-performance key-value store (a relative of
LMDB) that gets a lot of its speed by using memory-mapping and avoiding
copying. So I want to avoid copying the bytes.
The only collection I've found that lets me do this is `openarray`, and I've
found the functions that let me create one from a `cstring` and a length, but
the manual says `openarray` can only be used as a parameter. That doesn't seem
to be enforced, however: I can declare an object type containing an `openarray`
without errors.
I'm thinking of doing something like this:
type Transaction* = ref object
...
type Result* = object
bytes*: openarray[byte]
owner: Transaction
proc get(t: Transaction, key: string): Result =
...
Run
The `owner` field of the `Result` holds onto the `Transaction` object, keeping
it alive so the `bytes` remain valid. (The C API only guarantees the
availability of the data during the transaction it was accessed in.)
Is this OK, or something that could cause trouble?
—Jens
Re: Question about type safety and OOP
> `var a:seq[A] = @[B(val:1), B(val:2)]` > can not work as Nim is a statically
> typed language, seq[A] and seq[B] are different types, so assignment is not
> allowed.
There's nothing about static typing that forbids this; the collection class
just needs the appropriate generic assignment operator. I thought C++'s
`vector` allowed it, but I just tried it and it doesn't. But Swift's `Array`
does, so this compiles:
// This is Swift 5.2
class A { }
class B : A { }
var a: [A] = []
var b: [B] = []
a = b
Run
In Nim I think you could write a generic proc to assign a `seq[B]` to a
`seq[A]`, using the restriction `when B is A`.
Re: Question about type safety and OOP
I’m a Nim newbie, haven’t used seq much, and I’m making the assumption that seq is a reference type, ie. passed by reference not by value. If that’s wrong, ignore this!) You can’t assign an instance of seq[B] to a variable of type seq[A], because you could then use that variable to add a non-B object to the seq. Basically, seq[A] has looser constraints about what it can contain than seq[B] does. By analogy: I make a scrapbook of photos of cats. You come over while I’m out, pick up the book and say “oh, it’s a book of animal pictures! I’ll add this cute photo of my pet snake.” Then I find the photo later and am all “WTF is this snake doing with my cat pictures?!” Incidentally, this is only a problem if seq assignment is by-reference — otherwise it’s like you made a copy of my cat-photos book. Now you have your own animal-photos book and can paste snakes into it with no problem. It’s also only an issue if the variable assigned to is mutable — otherwise it’s like you pick up my book but have the courtesy to understand that you aren’t allowed to change it. (In type theory this kind of issue is referred to as _contravariance_.)
Re: NIM Integration error when trying to generate dynamic library from macOS Catalina 10.15.4
This looks like a mistake in your Makefile — I think you need to remove `Makefile` from the right-hand side of your rules, because it’s feeding the Makefile itself to the build commands. I think the error comes from the linker trying to treat the Makefile as a library.
Re: How to implement observer (publish/subscribe) pattern?
I think you could also do it in OOP style by defining a Subscriber object with a ‘changed’ method, then having the real subscribers subclass that and override the method. (Requires ‘pragma multimethods’, I think? I have to admit I haven’t tried any real OOP in Nim yet! I’d try it out in a playground but I’m on my phone right now...)
Best Nim translation of C {void*, size_t} struct
I'm working on Nim glue to a C API that uses `struct iovec` a lot as a parameter type — a generic (pointer, length) tuple indicating a range of memory to read or write. IIRC, a Nim `openarray[uint8]` is passed the same way as an `iovec`: as the address of the first byte, then the length. So if I changed the `iovec` parameters in the bridged C declarations to `openarray[uint8]`, to make them easier for me to use, would that work? And just as importantly, is this something that can be counted on to work in the future, either because it's official or because too many projects do it already? Or is there an even better way to do it? —Jens
Re: How mature is async/threading in Nim?
> how does your C++ architecture share data between threads? Carefully ;-) The Actor library doesn't restrict what parameters you can pass, but by convention we're careful to limit it to primitives, pass-by-copy objects (like std::string), immutable ref-counted objects, and Actor references. Or else we use rvalue references to enforce 'move' semantics. But mistakes happen, so I'm looking forward to a system that will have better enforcement.
Re: How mature is async/threading in Nim?
> Threads are just kind of hard to use with gc:refc, that is why gc:arc is > getting worked on. I'm still confused as to which GC is the default. I thought it was ARC, but what I'm reading recently makes it sound like ARC isn't quite stable enough yet. > Just regular threads are probably better if you are just writing and reading > from a single websocket connection. Async and threads are orthogonal, not opposite choices. What interests me about async/await is how it cleans up the control flow in source code, avoiding "callback hell". Again, I'm considering porting a large existing project, not writing something from scratch. I already know the requirements and the architecture. Nim is different from C++ but that doesn't mean I'm going to change the design to single-threaded. I definitely need multiple threads because the WebSocket messages invoke tasks that may be CPU-intensive or just block a while (database calls.)
Re: How to instantiate `ptr object`
> You can use closures or pass the data parameter as an input to the function. In some sense closures and objects are isomorphic, as are functional and OOP. There’s a famous old _koan_ about this: > The venerable master Qc Na was walking with his student, Anton. Hoping to > prompt the master into a discussion, Anton said “Master, I have heard that > objects are a very good thing — is this true?” Qc Na looked pityingly at his > student and replied, “Foolish pupil — objects are merely a poor man’s > closures.” > [(continues...)](https://citizen428.net/blog/of-closures-and-objects/) —Jens
Re: How mature is async/threading in Nim?
> it requires an introduction that explains to users what ARC is, how to make > use of it +1 The existing documentation is **great** (I’ve read the tutorial, manual, and “Nim In Action” cover to cover), but in some areas seems to lag behind. Which is understandable since the language is evolving quickly. I’m one of those weird people who likes writing documentation, so maybe when/if I get up to speed on this stuff I can help out.
Re: How mature is async/threading in Nim?
> Async does not mesh well with threds. Could you explain why not? My understanding is that it’s thread-agnostic; an unfinished async call is just a sort of lightweight continuation that can be restarted in any context. > Multiprocessing is more scalable anyways. This project is a library for (primarily) mobile apps, so that’s not an option!
Re: How mature is async/threading in Nim?
Right after posting that (how often this happens!) I came across [the big ARC thread](https://forum.nim-lang.org/t/5734) here from last December. It sounds like ARC means a lot of (positive) changes to the things I've read earlier, like: > The heap is now shared as it's done in C++, C#, Rust, etc etc. A shared heap > allows us to move subgraphs between threads without the deep copies but the > subgraph must be "isolated" ensuring the freedom of data races and at the > same time allowing us to use non-atomic reference counting operations. How to > ensure this "isolation" at compile-time was pioneered by Pony and we can do > it too via our owned ref syntax I've been interested in the Pony language for several years and adopting its memory model would be amazing! I'm still reading through this long thread. At this point I'm unclear on how much of this stuff is solid and enabled-by-default (in particular, what's the difference between "arc" and "orc"?)
How mature is async/threading in Nim?
The project for which I'm evaluating Nim does a lot of concurrent network I/O (over WebSockets) and database access. It needs to run on most mainstream platforms (iOS, Android, Mac, Windows, Linux.) The current implementation is in C++17, using a homemade Actor library. The other language I'm seriously considering is Rust. Nim seems to have the necessary pieces, like the `async` macro and the `asyncnet` module. But are they mature enough to use in production code? I have a few specific questions: 1. The asyncnet module documentation has a couple of caveats about Windows, like "on Windows it only supports select()" and "In theory you should be able to work with any of these layers interchangeably (as long as you only care about non-Windows platforms)." I'm not clear on how this would impact clients, and whether these caveats apply to using asyncnet or just the lower-level modules. 2. The async examples I've seen run on a single thread. Is there any support for distributing async calls across a thread pool? 3. The `spawn` function's thread safety seems to be based on segregating heap objects per thread (Erlang-style.) This can involve a lot of copying, especially when passing data buffers around. Is this something baked into the language or is it specific to the `spawn` function? Are there alternatives to this, like the more Rust-style model using move semantics? We found [an interesting quote on r/nim](https://www.reddit.com/r/nim/comments/giaeev/what_are_the_biggest_weaknesses_of_nim_in_your/fqdlmhf/) from a few days ago: > What ... caused my last team to abandon Nim in favor of Haskell ... was the > weak concurrency story. There is _a_ story there (thread-local heaps) but we > found it far too easy to get yourself into very confusing situations where > behavior wasn't as expected (especially since you can pass around pointers > which breaks the thread safety checker). To be fair, our C++ code has almost no built-in thread safety at all; you have to be careful what types you pass to Actor methods. But that's one of the things I'd like to improve on! I've written some prototype code in both languages (C API bindings, not any async or net code yet) and I have to say I really enjoyed Nim a lot. Rust I found frustrating, and ugly due to lack of function overloading and the need to convert between equivalent types like str/String. Nim also builds faster, and seems to generate much smaller code. But Rust does have huge momentum behind it so it feels safer for that reason, and the ironclad memory safety is a good thing to have. Any comments or perspective from those who've been using Nim a lot? \--Jens| ---|---
Re: Revisiting my oldest Nim project.
> floats will cause issues with rounding or difference wrt roundtrips, not to > mention stringification I don't think the same issues apply to timestamps. They're not user-visible in raw form, and you're not usually working with adjacent timestamps that are nanoseconds apart, so it doesn't matter if they're off by some tiny epsilon. I like the fact that you never have to worry about overflows. And treeform's point about lack of int64 in JS is important — a lot of JS parsers, in various languages, just parse _any_ number to float64. C++'s std::chrono module does support arbitrary backing types for times (it's a template parameter) but IMHO that contributes to making the library super awkward to use; you have to invoke a manual conversion function to cast times to different units.
Re: Experimenting with a FreeRTOS OS Port
This is great! ESP32 is a nice platform; I've used it a bit in the past, with C++. It'd be good to have Nim running on it. I don't have the time to contribute right now; just offering moral support and thumbs-up
Re: Revisiting my oldest Nim project.
> For a time library, the lowest building block should be the timestamp of a > single float64, not a complex calendar object. You should store timestamps > and transfer timestamps. Yup. In units of seconds since 1/1/1970, right? Although I've sometimes worried how soon the precision will degrade, since there are 'only' 52 bits of mantissa. There are cases where you want to store a (timestamp, timezone) tuple, like for blog posts, when it's useful to know what the local time was.
Re: Revisiting my oldest Nim project.
> Bummer, I still like them better than json. :-) > I wrote one, called Fleece > <[https://github.com/couchbaselabs/fleece>](https://github.com/couchbaselabs/fleece). > It's used in the Couchbase Lite database, because it's faster than JSON. > There's now [a Nim binding of > it](https://github.com/couchbaselabs/couchbase-lite-C/blob/master/bindings/nim/src/CouchbaseLite/fleece.nim) > ... it's part of the experimental Couchbase Lite binding I did over the past > week. It's not easily separable yet, and it requires the C++ library, but > that can be cleaned up.
Nim support for CodeRunner app
CodeRunner <[https://coderunnerapp.com](https://coderunnerapp.com)> is a really useful utility app for Mac, a GUI scratchpad editor for writing and running code in dozens of languages, from Python and Bash to C/C++. It's not free or open-source, but IMHO it's well worth the $15. I use it a lot. I put together a Nim plugin for it. Mostly I just needed to write a little shell-script to invoke the Nim compiler. The syntax-coloring library the app uses already supports Nim, but there's no syntax-aware indentation or autocomplete. Still, I find it really useful for trying out snippets as I learn the language. To install: * Download <[https://mooseyard.com/tools/Nim.crLanguage.zip](https://mooseyard.com/tools/Nim.crLanguage.zip)> * Unzip it * In CodeRunner's "Run" menu choose "Edit Languages..." * Pull down the gear menu at the bottom of the window and choose "Import..." * Select the "Nim.crLanguage" file from step 2 \--Jens| ---|---
Re: New blog, with some Nim articles
Once Nim can intelligently add `restrict` to its generated C when it knows pointers are not aliased, it should be able to beat C ... because no C/C++ programmer I know of ever uses `restrict`, it's too confusing 藍 (I say "once" because I saw an issue in the Nim bug tracker calling this out as work that hasn't been done yet.)
Nim version release notes?
I’ve been using Nim 1.2 so far, because it’s what Homebrew poured me, and because I figured it’s best to start with a stable release. But a lot of devs seem to use 1.3.x ... and I’m curious what new goodies it has. But I can’t find release notes or a what’s-new list anywhere. Is there such a thing? Or should I try diffing the nightly docs against the 1.2 ones? Or if someone just wants to reply and talk about their favorite new feature or improvement, that’s cool too :) —Jens
Re: How to parse html wild?
Browsers have _always_ supported “tag soup” HTML, back to Mosaic and Netscape. Unless the content type is XHTML, you cannot expect any sort of valid structure. For parsing “wild” HTML, preprocessing through some widely-used tidier is probably the best bet, since its interpretation of bad markup is hopefully similar to a browser’s.
Re: How to instantiate `ptr object`
I wasn’t aware of that book, though I’ve been guzzling from the Nim-documentation firehose for a week now. You should get it linked from the Nim-lang “learning” page!
Re: A good word for idiomatic nim?
"nimsy". As in "All nimsy were the borogoves..."
Q: An object variant case with no field?
I'm implementing a variant object type, where one of the enum cases has no fields associated with it, but get a syntax error: type AuthenticatorType = enum None, Basic, Session, Cookie Authenticator = object case type: AuthenticatorType: of None: # Error: identifier expected, but got 'keyword of' of Basic: username, password: string of Session: sessionID: string of Cookie: name, value: string Run What can I put after `of None:` to satisfy the parser? Or is it impossible to have a case with no associated value? (I've checked [the manual](https://nim-lang.org/docs/manual.html#types-object-variants) but it says nothing about this possibility.)
Re: From seq[] to C array, the safest way...
> It's not a hack, I don't know where it is documented, but This openarray usage with C bindings would be useful for the API I’m working with; but I’d need to see it documented. If it’s not documented then it’s just a side effect of the implementation, which could change at any time. I suppose this is another way of asking whether Nim’s C backend has a documented ABI.
Best practices for wrapping opaque C pointer types?
I'm wrapping a C API that uses the typical fake-OOP idiom of opaque reference types as "classes". In this case they're ref-counted. For example: typedef struct Box Box; Box* box_new(int size); void box_release(Box*); void box_retain(Box*); int box_get_size(const Box*); Run I can easily create a Nim API for this, by hand or by using c2nim, and it's pretty friendly to use, except that now the Nim programmer is responsible for memory management, balancing every box_new with box_release. Not good. I dug around and found the "Nim Destructors And Move Semantics" document, which describes the `=destroy` and `=` hook functions. So now I've made an object type that holds a C pointer, with a destructor and a copier. type NimBoxObj = object handle: ptr Box type NimBox = ref NimBoxObj proc `=destroy`(b: var NimBoxObj) = release(b.handle) proc newBox(size: int): NimBox = NimBox(handle: box_new(size)) Run I'm just wondering if this is the best approach. It's got a few flaws: * Another heap allocation for each C object * I have to redeclare the entire API (minus retain/release) for my new NimBox type I thought of doing this without the refs — just use NimBoxObj directly — making it more like a C++ smart pointer type. But that means NimBoxObj instances will get copied a lot during assignments and function calls, doesn't it? I'm concerned that the consequent calls to `box_retain` and `box_release` might be performance issues. Or am I making too much of this? Ideally there'd be a way to avoid wrapping the C type in an object, but I suspect that won't work because it's a `ptr` and that does explicitly mean _[unmanaged](https://forum.nim-lang.org/postActivity.xml#unmanaged). Any suggestions? Or pointers [sic] to existing libraries that do a good job of this?
Re: Some problems trying to eliminate 'nil'
> You are not supposed to use not nil types as Option generic parameters Then how do I make a function that conditionally returns a value of that type, like the `makeFoo` function in my example? "Option(al)" is the canonical way to do this in other languages. I could use a regular ref as the parameter, like `Option[ref FooObj]`, but then the value I extract from such an Option can't be assigned to a regular Foo since the compiler doesn't realize it can't be nil... I can understand why the current implementation of the `Option` class might have trouble with a not-nil type, since the pointer it stores internally needs to support nil. That seems more like a bug/limitation of `Option`. Sorry to hear that `not nil` is less capable than it looks (it really should be moved out of the main compiler docs), but I'm glad there's some recent design work going on. IMHO this is one of the weak corners of Nim as compared with other modern languages like Rust and Swift.
