Re: Why is my HTTP handler not gcsafe?
btw, if you need to pass some state to callback in server(), wrap it will closure. Nim current approach force threads to talk explicitly. I recall my memory that I have written a database connection pool and logger for a web server, which they are supposed to be globally accessible. And then I have to fight with the unexpected gcsafe for a long time... dealing with threads in nim is another story, I encountered unexpected exits without any messages if I did something wrong in ITC...
Re: Why is my HTTP handler not gcsafe?
In short, no declaration of non-primitives global variable. There seems
something declared at module level in blip that you didn't show.
When a proc is annotated with gcsafe, any proc called inside the proc has to be
gcsafe also, regardless of whether you are compiling with --threads. And if any
proc marked as gcsafe ever touched a module level non-primitive variable. An
error will throw. The first proc annotated with gcsafe here is the callback
function in serve().
Other people on this forum will probably explain better me. But here my 3-yo
attempt, if a proc is annotated with {.gcsafe.}, the proc is supposed to be run
in multi-threaded environment. And in current nim memory management model, each
thread has its own gc and they work on their own, no knowledge of each other.
This raises a problem: which gc should deal with the shared memory allocation
and freeing? Nim current answer is to avoid the problem at its first place: not
allow existence of non-primitives global variable touched at compile time.
gcsafe was one of the most annoying in my nim-experience... I had a long time
didn't know why my single threaded web server has to be complained with
non-gcsafe. Generally speaking, avoid global variable at all, this make testing
easier and common practice also.
Re: Transform procs at compile time?
Another part of answer: * static * repr * writeFile
Re: Why is my HTTP handler not gcsafe?
I've added some more code, and now I'm getting actual errors that seem to
explain the cause of the GC-safe violation ... if I could understand them.
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(318, 24) Warning:
'receiveLoopNimAsyncContinue' is not GC-safe as it accesses
'nameIterVar`gensym21340263' which is a global using GC'ed memory [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/Users/snej/Code/nim/blip/src/blip.nim(126, 6) Warning: 'receiveLoop' is
not GC-safe as it calls 'receiveLoopNimAsyncContinue' [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/Users/snej/Code/nim/blip/src/blip.nim(136, 6) Warning: 'run' is not
GC-safe as it calls 'receiveLoop' [GcUnsafe2]
/Users/snej/Code/nim/blip/src/server.nim(14, 25) template/generic
instantiation of `async` from here
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(278, 31) Warning:
'cbIter' is not GC-safe as it calls 'run' [GcUnsafe2]
Run
`receiveLoop` is pretty simple (see below), but `receiveLoopNimAsyncContinue`
and `nameIterVar`gensym21340263` must be internal functions created by the
macros behind the `{.async.}` pragma, so I don't really have a clue what's
going on...
proc receiveLoop(blip: Blip) {.async.} =
while blip.socket.readyState == Open:
let packet = await blip.socket.receivePacket()
case packet.kind
of Binary: blip.handleFrame(cast[seq[byte]](packet.data))
of Close: return
else: continue
Run
Re: Why is my HTTP handler not gcsafe?
...also, at a higher level, why is gcsafe even an issue at all, when all this async stuff is supposed to be running on the same thread?!
Re: Trivial AsyncHttpServer example crashes on every request
This is a known problem that was fixed and backported to 1.2.0. If you are using choosenim delete and re-install 1.2.0 problem will be solved.
Why is my HTTP handler not gcsafe?
I've now got some slightly less trivial async code I'm trying to get to run,
using the 'news' WebSockets module. The compiler complains that my HTTP handler
isn't gcsafe, and suggests "Annotate the proc with {.gcsafe.} to get extended
error information."
However,
1. I can't see how my code isn't GC-safe, since it has no global variables
whatsoever.
2. If I add the `gcsafe` pragma to my callback proc, it now compiles without
errors! Where's my "extended error information"?
import blip
import news, asyncdispatch, asynchttpserver
when isMainModule:
echo("Starting Blip server on port 9001...")
var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
if req.url.path == "/blipsync":
var ws = await newWebsocket(req)
echo "Creating new Blip"
var blip = newBlip(ws)
await blip.run() # <-- this line triggers the gcsafe error
echo "...Closed Blip"
await req.respond(Http404, "Nope", newHttpHeaders())
waitFor server.serve(Port(9001), cb)
Run
If I take out the line `await blip.run()`, the error goes away. But neither
that function nor anything it calls appears to involve any global state. (It's
too much code to post here in the forum, unless someone really wants me to!)
Did adding the `{.gcsafe.}` pragma silence a false alarm? Or is it suppressing
a genuine problem that will cause thread-safety issues?
Thanks!
—Jens
Re: Trivial AsyncHttpServer example crashes on every request
I confirmed that if I change line 5 to await req.respond(Http200, "Hello World", newHttpHeaders()) Run (passing in a non-nil headers object), the program works fine. I guess this has turned into a bug report, although when I started this post I didn't know what the problem was. Thanks for the rubber-duckie debugging, forum! What Github repo should I file this under? Nim itself?
An opportunity to give Nim some exposure - Gamesfromscratch starts covering programming languages
Over on Youtube, there's a relatively big channel called Gamesfromscratch. So far, the guy behind it mostly covered game engines, but he has recently made several videos about gamedev-relevant programming languages. Today, he posted a poll asking if he should continue covering languages and if so, which ones. The poll is here: [https://www.youtube.com/post/UgzILZp-hmgi9Jn3Azx4AaABCQ](https://www.youtube.com/post/UgzILZp-hmgi9Jn3Azx4AaABCQ) This could be an opportunity to give Nim some time in the spotlight, so smash that "yes" button and demand Nim coverage in the comments :)
Trivial AsyncHttpServer example crashes on every request
I'm exploring WebSockets and async/await. But I'm finding that the most trivial
example of an async HTTP server crashes on every request. If I run an exact
copy of the example from the asynchttpserver documentation:
import asynchttpserver, asyncdispatch
var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
await req.respond(Http200, "Hello World")
waitFor server.serve(Port(8080), cb)
Run
... it will crash hard when I send an HTTP GET to it, from any client I've
tried (Safari, curl, httpie):
/Users/snej/Code/nim/blip/src/server.nim(7) server
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(1886) waitFor
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(1576) poll
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(1340) runOnce
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncdispatch.nim(210)
processPendingCallbacks
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(37)
processRequestNimAsyncContinue
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asynchttpserver.nim(258)
processRequestIter
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(319) cb
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asyncmacro.nim(34)
cbNimAsyncContinue
/Users/snej/Code/nim/blip/src/server.nim(5) cbIter
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/asynchttpserver.nim(103) respond
/usr/local/Cellar/nim/1.2.0/nim/lib/pure/httpcore.nim(180) hasKey
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Run
I don't know this code at all, but from quick inspection I can see that the
`respond` function in asynchttpserver.nim has a `headers` parameter that
defaults to `nil`, but it calls `headers.hasKey("Content-Length")` without
checking for `nil` first. Oops.
Is this module generally considered solid, and this is just an unusual bug
where no one ever tested the examples in the docs? Or is there another module I
should use instead?
—Jens
Re: How does one use locks (need a brief example)
Kinda pseudocode, fragments scavenged from one of my projects: import locks var lock: Lock myflag: bool lock.initLock() proc foobar() = echo "before lock" lock.withLock: if myflag: echo "...do something with myflag..." # ... work on myflag more ... echo "after lock" Run
Re: Unicode support for Windows 10 console
Codes have been updated again. * In **console.nim** initial buffer length was set to **2048** Utf16Chars in order to avoid unnecessary relocations. * In **console2.nim** , **consoleReadLine** was trimmed down to what **readConsoleW** actually supports. @Araq Do you prefer the **fgetws** way or the **readConsoleW** way? * The first one fits in with standard C library, but is a little messy. * The second one is cleaner, but adds an extra variable to mark **EOF** and does not fit in well with the **feof** way, so it may not be a good candidate if the relevant functionality is to be merged with that of **readLine**.
Re: Transform procs at compile time?
I think that it should be more easily done from NimScript, it seems that macros module is supported in NimScript. I think I can write a script that processes modules to a temp folder and then compile them from NimScript. What do you think?
Re: Nim's popularity
This may seem pretty boring but I think Nim's killer use-case is interfacing with C++, to my knowledge no other language outside of D comes close. A macro driven glue that uses compile time introspection to generate bindings to C++ from Nim in addition to bindings to Nim from Language X will open up a ton of possibilities for people who want to use X but need things from C++. Nim can be that glue, especially with arc/orc users of X don't even have to spin up a runtime and take it down. Having written C++ bindings the hard way this is a huge albeit unglamorous opportunity to get into a lot of shops. Instead of "rewrite it Rust" you "reuse it with Nim".
Re: Transform procs at compile time?
I don't quite get it. For example I have a few modules that I want to process. I will need to use staticRead for each file, use parseStmt to get the AST, transform it and what do I do next? Do I need to save this modules to a temp folder using astGenRepr and add them to the compilation pipeline? Can I write a macro that is when called from whithin a module, reads code of that module, rewrites AST and replaces original AST with rewritten one or something of that kind?
Re: Transform procs at compile time?
partial answers: * parseStmt + staticRead * [https://github.com/nim-lang/Nim/pull/9560](https://github.com/nim-lang/Nim/pull/9560)
Re: Nim's popularity
Gracias @juancarlospaco. This is exactly what I was looking for. @treeform, nice overview. Given Nim's potential, I am surprised about point 1. I think/hope this is a matter of time.
Transform procs at compile time?
Is there a way to get the AST for all procs of a module and apply some transformation on it at compile time. I've been looking at this [https://github.com/Yardanico/nim-strenc/blob/master/src/strenc.nim](https://github.com/Yardanico/nim-strenc/blob/master/src/strenc.nim) \- here the code uses term rewriting macros to apply transformations to all string literal in the module. But I can't figure out if or how it can be applied to procs. Any ideas? Or maybe there's a better way to solve this?
Re: Awk-Style text processing with Nim
same remarks as
[https://github.com/nim-lang/Nim/issues/14637#issuecomment-644338428](https://github.com/nim-lang/Nim/issues/14637#issuecomment-644338428)
the standard nim version is:
* more readable
* syntax highlights correctly
* interacts correctly with the rest of the language
* leads to code that's more reusable
* is correctly indented instead of relying on features only invented to
support SCF (end if|case|for|etc)
* supports block comments
etc...
If you have boilerplate, just use templates+macros, no need to bake in a
sublanguage into nim. SCF is a misfeature that needs to be deprecated.
import strformat, streams, parsecsv
var x: CsvParser
open(x, newFileStream(stdin), "-", '\t')
template p(a: string) = echo &(a)
p """
My Report
=
{"":-<20} {"":-<20}"""
while x.readRow():
p """{x.row[0]:<20} {x.row[1]:>20}"""
p """{"":-<20} {"":-<20}"""
Run
Re: Nim's popularity
@BugFix, I am also an AutoIt lover and user. And i have the same feeling about data types. AutoIt needs at least an easy way to declare a struct and a container data type to hold that struct. OOP is not in their list. I wish i could override the autoit message loop in order to handle more window messages.
Re: Nim's popularity
[https://tjpalmer.github.io/languish/#y=mean=nim%2Ccrystal%2Craku%2Chaxe%2Celm%2Cd%2Cwebassembly%2Cpurescript%2Cracket%2Cllvm%2Cpony%2Cred%2Cgdscript%2Cmathematica%2Cvisual+basic%2Cprocessing](https://tjpalmer.github.io/languish/#y=mean=nim%2Ccrystal%2Craku%2Chaxe%2Celm%2Cd%2Cwebassembly%2Cpurescript%2Cracket%2Cllvm%2Cpony%2Cred%2Cgdscript%2Cmathematica%2Cvisual+basic%2Cprocessing)
Re: Comparisons of Nim to the Chapel computer programming language...
Replying to @GordonBGood and @mratsim: > > There is no reason to have such a high overhead especially in high > > performance computing. OpenMP is way way way lower. > > I agree, and I'm still not 100% sure where all the time is going Are you able to share the benchmark you were using with us (as well as information like back-end compiler and version, processor, etc)? In our experiments, Chapel has generally shown to be competitive with OpenMP, so it would be interesting for us to understand better what you were doing (prior to resorting to a homegrown thread pool) in order to make sure nothing's going horribly awry. I'd also be curious whether you were using CHPL_TASKS=qthreads or CHPL_TASKS=fifo. Thanks. > I expect [Chapel's data pallelism] is similar to CoArray Fortran Chapel's data parallelism is significantly different than Co-Array Fortran, where an array-of-arrays approach is taken for distributed arrays. In contrast, Chapel's data parallelism is based on global-view domains (index sets) and arrays, which are an evolution of concepts that were pioneered by ZPL in the 1990's. By default, most data parallelism in Chapel is implemented using #cores tasks where cores is the number of processor cores to which the index set or array are distributed. This ensures that the computational granularity is based on the hardware parallelism rather than the data collection (though programmers can override these defaults if they want something finer/coarser). > I can see that it could be of interest if one had access to such a machine or > group of machines, which most of us probably never will. For those not in the market for a Cray or commodity cluster, I suspect AWS, Azure, and Google Cloud would be happy to offer you such access for a reasonable fee. :)
Re: Nim's popularity
This talk has a pretty good part about how most popular languages got popular: [https://www.youtube.com/watch?v=QyJZzq0v7Z4](https://www.youtube.com/watch?v=QyJZzq0v7Z4) To summarize, according to the talk, there are 5 major ways to popularly (top 10): 1. Killer App: C, Ruby, PHP 2. Platform Exclusivity: JS, Obj-C, Swift, C# 3. Quick Upgrade: C++, Kotlin, TypeScript 4. Epic Marketing: Java 5. Slow & Steady: Python I think Nim right now is doing a little bit in 3 out of 5 directions. 1\. Killer App: Nim has the ability compile to both JS and Native allowing same code. If only Nim could do this it would be really cool, but Node/JavaScript can do this as well. Nim needs to find some thing else. Nim can interface between c/c++ and other languages. The FFI is really good. Is this enough for a killer app? 2\. Platform Exclusivity: Nim is not doing anything here. There is no only Nim platforms and probably never will be. 3\. Quick Upgrade: Nim is kind of a quick upgrade from Python and maybe C++. But it does not feel quick "enough", but it's there. Languages like Kotlin and TypeScript are much quicker. 4\. Epic Marketing: Nim is doing nothing here. No one is spending a million dollars on Nim ads or Nim conferences. 5\. Slow & Steady. The slow community path. This is probably where Nim is growing the most right now. But not sure its fast enough. I also think that most languages have this path as well. This path is way too crowded.
Re: Nim's popularity
I would like to tell my way to Nim. I have been writing smaller programs every now and then for more than 30 years. Usually these are automations or add-ons. In the past few years I have written a lot with the AutoIt scripting language, which in development now goes far beyond the stage of pure automation. But with increasing experience and knowledge, some things are no longer an advantage (e.g. data type: Variant). I also lacked object-oriented programming. Since I also have experience with Phyton and Lua, I looked for a language with a similar syntax. OOP and creating executables were other criteria. During my internet research I came across Nim and after reading the basics I fell in love with the language. Of course, this is different from a scripting language and for me it is about learning and learning again. Now when I see my first steps and, in contrast, look at the material that is being discussed here in the forum, I feel like a school boy again. Even though I am almost 60 years old, I still enjoy programming - and learning.
Nim's popularity
I was wondering (just out of curiosity) about current Nim's popularity. I know every metric is somehow biased, but I would like to get a feeling about how is evolving. I was looking as well, for the biggest projects developped in Nim (apart from Nim itself). I bet there is a way to get that information from Github, but I don't know how. It looks this is quite a recurring topic. I have searched about it and I found (far from complete): * 2017: [Nim 129th in the Tiobe index](https://twitter.com/nim_lang/status/830398335026659328?lang=es). * [Nim hitting 800 modules](https://www.reddit.com/r/nim/comments/9rf1ob/happy_800_modules_nimbles_ytd_growth_beats_d_c/) * 2018: [Nim 43th most starred language](https://madnight.github.io/githut/#/stars/2018/3). (It keeps on being at that level: [44th in 2020Q1
Re: Comparisons of Nim to the Chapel computer programming language...
Don't transport data, transport programs. This is the [motto of Hadoop](https://www.google.com/search?sxsrf=ALeKk03SEWZNtovL4re3tWEED-CW1UYjyw%3A1592237139427=U5znXtHTGdjQ0PEPgtyI-AY=move+code+to+data=move+code+to+data_lcp=CgZwc3ktYWIQAzIGCAAQBxAeMgIIADIGCAAQCBAeMgYIABAIEB4yBggAEAgQHjIGCAAQCBAeMgYIABAIEB46BwgAEEcQsANQ1aABWNWgAWDMqgFoAHAAeACAAVCIAZMBkgEBMpgBAKABAaoBB2d3cy13aXo=psy-ab=0ahUKEwiRyd_ImYTqAhVYKDQIHQIuAm8Q4dUDCAw=5), slightly rephrased. What does it mean practically for programmers? I think @mratsim has a point (from experience at a bio startup) that we'll see increased need for NUMA aware APIs. I could be wrong as well, but it seems that it's one natural avenue for getting better performance post Moore's law. Great stuff, thanks for the writeup on Chapel @GordonBGood! It's a fascinating language.
Re: Can't access fields of object returned by a procedure
For oldcommers too! Error should say x is not public, not that x is missing.
Re: Comparisons of Nim to the Chapel computer programming language...
> In all languages (Go, Nim, ...) channels are built either on top of > Lock+Condition variables or Atomics (reusing queue designs) but I expect we > will reach the limits of memory coherency at the CPU level and that in the > future NUMA will become increasingly important and we might see hardware > channels being more popular (rather than restricted to stuff like Infiniband) Well this has been claimed for over ten years now, it's one reason why Nim did bet on thread local heaps + message passing and I don't see it becoming the reality anytime soon. I can be wrong of course. However, it is insightful to look at what GPUs do. The classical view is that they are based on shared memory. But when you look at it from a different angle what happens is that inside a GPU a program fragment is passed over to the data. Maybe I'm nuts and is has nothing to do with GPUs, but the idea is a good one. Don't transport data, transport programs.
Re: Comparisons of Nim to the Chapel computer programming language...
> Difference of opinion, Golang seems to do as is your opinion, Haskel builds > everything on MVar's, and F# has both async and task parallelism with the > Task type it gets from DotNet including the FlowVar concept, but with those > Task's able to work together with Async. F# doesn't have built-in channels, > so if required on would have to build them from primitives. > > My opinion could be changed if the version of FlowVar's I propose truly can't > work with a version of async, but I'd like to see the implementation as by F# > researched first. The main reason why I think channels should be a primitive building block is that it can be implemented in hardware on message-passing-based hardware or cluster-on-chips: * Intel SCC had hardware channels [https://en.wikipedia.org/wiki/Single-chip_Cloud_Computer](https://en.wikipedia.org/wiki/Single-chip_Cloud_Computer) * I assume Tile CPUs from Tilera/EzChip/Mellanox/Nvidia (successive acquisitions by a bigger fish) also have hardware channels [http://web.mit.edu/6.173/www/currentsemester/handouts/L18-tilera-multicore.pdf](http://web.mit.edu/6.173/www/currentsemester/handouts/L18-tilera-multicore.pdf) * Similarly for Adapteva 1024 cores RISC network-on-chip CPU: [https://www.parallella.org/docs/e5_1024core_soc.pdf](https://www.parallella.org/docs/e5_1024core_soc.pdf) * You can reuse Infiniband or MPI channels as primitives for distributed computing. Adding async support on top of a channels only requires adding a way for the channel to return control the event loop. If channels use `tryRecv` and `trySend` it's easy to add a `poll()` or equivalent callback to the event loop. In all languages (Go, Nim, ...) channels are built either on top of Lock+Condition variables or Atomics (reusing queue designs) but I expect we will reach the limits of memory coherency at the CPU level and that in the future NUMA will become increasingly important and we might see hardware channels being more popular (rather than restricted to stuff like Infiniband)
Re: Exporting string functions from DLLs
As @Javi alluded to NimMain is what sets up the GC. So without calling NimMain the GC won't work and thus no Nim strings as they require the GC. It's worth noting though that \--noMain only removes the call to NimMain, the procedure is still generated. So you can import it into your Nim code as any other C function and call it yourself.
Re: Unicode support for Windows 10 console
Nice to see progress here. Maybe we can improve the stdlib's solution too, PRs are welcome.
Re: Can't access fields of object returned by a procedure
Anytime soon now (ha!) we'll improve the error message for this, it's a common gotcha for newcomers.
