Re: Best Nim translation of C {void*, size_t} struct
I would use `openarray[uint8]` too. Strictly speaking though Nim uses a Nim `int` (== `ssize_t`) as the length information and so it's incompatible for arrays which contain more than 2 billion elements (on a 32 bit machine). Never happens.
Re: Faster and Safer Raytracing in Nim
Very good, this debunk an old blog article that show slowest results compared to C++
Re: GitHub Actions: workflow to cross compile and release to GitHubRelease
It's on my list to learn GitHub Actions. I just had a comment about the NimScript task archive. You can put that into a global config.nims of yours which you pull in before building all of your projects. I have started using [this minimal config.nims](https://github.com/kaushalmodi/version/blob/master/config.nims) in my new projects. The project's [.travis.yml then pulls in](https://github.com/kaushalmodi/version/blob/09100e7d619e8b681bb794d7b38d2b66e23a8af9/.travis.yml#L60) my [global config.nims](https://github.com/kaushalmodi/nim_config/blob/master/config.nims) which enables running my generic tasks using nim docs, nim musl .. and nim test.
GitHub Actions: workflow to cross compile and release to GitHubRelease
I wrote a workflow to cross compile commands and release to GitHub Release in
GitHub Actions.
[https://github.com/jiro4989/repo-template-nim/blob/master/.github/workflows/main.yml](https://github.com/jiro4989/repo-template-nim/blob/master/.github/workflows/main.yml)
name: build
on:
push:
pull_request:
release:
types: [published]
env:
APP_NAME: 'APPNAME' # TODO: need change
jobs:
skip:
runs-on: ubuntu-latest
steps:
- run: echo "Skip job"
before:
runs-on: ubuntu-latest
if: "! contains(github.event.head_commit.message, '[skip ci]')"
steps:
- run: echo "not contains '[skip ci]'"
build:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- ubuntu-latest
- windows-latest
- macOS-latest
needs: before
env:
NIM_VERSION: 1.2.0
steps:
- uses: actions/checkout@v1
- name: Cache choosenim
id: cache-choosenim
uses: actions/cache@v1
with:
path: ~/.choosenim
key: ${{ runner.os }}-choosenim-${{ env.NIM_VERSION }}
- name: Cache nimble
id: cache-nimble
uses: actions/cache@v1
with:
path: ~/.nimble
key: ${{ runner.os }}-nimble-${{ hashFiles('*.nimble') }}
- uses: jiro4989/setup-nim-action@v1.0.2
- run: nimble build -Y
- run: nimble test -Y
- name: Archive files
run: nimble archive
- name: Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v1
with:
files: 'dist/${{ env.APP_NAME }}_*.*'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
create-tag-draft:
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/master'
needs: build
steps:
- uses: release-drafter/release-drafter@v5.3.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Run
This workflow creates a tag-draft when master was pushed.
And when you published the tag-draft, this workflow do cross-compiling your
commands and release to GitHub Release.
[https://github.com/jiro4989/repo-template-nim/releases](https://github.com/jiro4989/repo-template-nim/releases)
`run: nimble archive` is a original nimble task.
This task creates a release file.
It creates .zip file when OS is windows.
It creates .tar.gz file when OS is MacOS or Linux.
[https://github.com/jiro4989/repo-template-nim/blob/master/APPNAME.nimble](https://github.com/jiro4989/repo-template-nim/blob/master/APPNAME.nimble)
import os, strformat
task archive, "Create archived assets":
let app = "APPNAME" # TODO: need change
let assets = &"{app}_{buildOS}"
let dir = "dist"/assets
mkDir dir
cpDir "bin", dir/"bin"
cpFile "LICENSE", dir/"LICENSE"
cpFile "README.rst", dir/"README.rst"
withDir "dist":
when buildOS == "windows":
exec &"7z a {assets}.zip {assets}"
else:
exec &"tar czf {assets}.tar.gz {assets}"
Run
Best Nim translation of C {void*, size_t} struct
I'm working on Nim glue to a C API that uses `struct iovec` a lot as a parameter type — a generic (pointer, length) tuple indicating a range of memory to read or write. IIRC, a Nim `openarray[uint8]` is passed the same way as an `iovec`: as the address of the first byte, then the length. So if I changed the `iovec` parameters in the bridged C declarations to `openarray[uint8]`, to make them easier for me to use, would that work? And just as importantly, is this something that can be counted on to work in the future, either because it's official or because too many projects do it already? Or is there an even better way to do it? —Jens
Re: Setting up a FreeBSD VM for Nim development
I'd be more than happy for it to become a blog post or wiki entry if it would be helpful.
Re: Faster and Safer Raytracing in Nim
I checked the schedule dynamic/static but it seems to be something else. If someone can reproduce that would be helpful. You probably need at least 4~8 cores.
Re: Setting up a FreeBSD VM for Nim development
it can even be a blog article. I find the debugging tool blog article much more pleasant to read in a blog post than in the wiki.
Re: Setting up a FreeBSD VM for Nim development
Would be nice to have this on the Wiki. :)
NIM Integration error when trying to generate dynamic library from macOS Catalina 10.15.4
Hello everyone I am trying to generate a dynamic library with integration of NIM and PostgreSQL, it works very well in Ubuntu, but in macOS it generates the following error ld: warning: ignoring file Makefile, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x50 0x47 0x5F 0x53 0x45 0x52 0x56 0x45 0x52 0x5F 0x49 0x4E 0x43 0x3D 0x20 0x24 ) Run Este es el make file con el que trato de generar la bibilioteca dinamica # Objetos de C MY_OBJS=addone.o magic.o utilities.o concat.o all: $(MY_LIB_NIM) $(MY_OBJS) $(MY_LIB_GCC) @echo Compilacion en Nim culminada con exito!!! $(MY_LIB_NIM): *.nim Makefile nim c -d:release --gc:arc -d:useMalloc --passC:-fPIC --noMain --opt:speed --app:staticlib --outdir:$(NIM_OUTDIR) --nimcache:$(NIM_CACHEDIR) --out:$(basename $<).a --header:$(basename $<).h $< %.o: %.c Makefile gcc -std=c11 -fPIC -O2 -D_GNU_SOURCE -c $< -I$(PG_SERVER_INC) -I$(NIM_INCLUDE_BASE) -I$(NIM_INCLUDE_LOCAL) $(MY_LIB_GCC): $(MY_OBJS) Makefile gcc -shared -fPIC -O2 $^ $(MY_LIB_NIM) -o $@ Run Any solution for this error? Thank you!
Re: Write Nim in Matlab/Julia style using macros while still deploy to Cloud/PC/GPU/embedded?
Thanks a lot for this information and for the Neo package! I'll take your advice.
Re: Write Nim in Matlab/Julia style using macros while still deploy to Cloud/PC/GPU/embedded?
Wonderful! Arraymancer is definitely doing more than I thought. I will go through the links carefully and try these features. Thanks!
Re: Faster and Safer Raytracing in Nim
So the parallelization of SmallPT uses `dynamic` scheduling #pragma omp parallel for schedule(dynamic, 1) private(r) // OpenMP for (int y=0; yhttps://www.cs.brandeis.edu/~dilant/WebPage_TA160/initialsllides.pdf](https://www.cs.brandeis.edu/~dilant/WebPage_TA160/initialsllides.pdf) image:: [https://gist.githubusercontent.com/mratsim/aaecdb8d77582bcfc2994b0ee66b99d5/raw/6cb890ee5ff68946133ba45fd9013ff555156fa2/2020-05-22_18-33.png](https://gist.githubusercontent.com/mratsim/aaecdb8d77582bcfc2994b0ee66b99d5/raw/6cb890ee5ff68946133ba45fd9013ff555156fa2/2020-05-22_18-33.png) A runtime without any load balancing wouldn't be able to scale a raytracing application (which makes it an interesting load balancing benchmark) I expect the GCC team broke dynamic scheduling and default to static. An easy way to confirm that is to change the schedule from dynamic to static, and rerun with GCC8 and Clang and see if the perf degradation matches GCC10. I leave that as an exercise to the reader (just joking).
Re: Faster and Safer Raytracing in Nim
Before we re-publish this as an article on our website, can you explain the 2min14.616s outlier for GCC 10 C++ OpenMP? Or can at least somebody confirm these numbers? :-)
Re: raise error using zig as cross compile
''' llvm-strip --strip-all gencfg or strip -s gencfg (utility from binutils) upx --best gencfg '''
Converting to another's variable type
I have a sequence of distinct types (despite all of them are actually int). I would like to "copy the types" into a new sequence. (Not looking for workarounds, because the problem is a little bit more involved) Something as the following: type A = distinct int B = distinct int C = A | B var d = newSeq[C](2) d[0] = 1.A d[1] = 1.B var e = newSeq[C](2) for idx, item in d: e[idx] = 5 # How do make "5" to be: `typeof(d[idx])`??? Run
Re: How mature is async/threading in Nim?
> Not really. What you can achieve is something similar to what I created with > httpbeast: each thread running its own Async event loop and allowing the > system to load balance the socket connections. For clients that will likely > be trickier. This _event-loop per thread_ is only required on Linux where epoll is inherently single threaded. Using IOCP you simply spawn N threads and make them spin on GetQueuedCompletionStatus. All balancing is done by the kernel. ARC should make this work more natural.
Re: raise error using zig as cross compile
No, you'll have to find strip for the architecture of the binary, zig cc doesn't expose that functionality - and again, it's not a Nim issue ;)
Re: Faster and Safer Raytracing in Nim
@mratsim, I agree with @aredirect: this repository of yours is pure gold!
Re: How mature is async/threading in Nim?
I didn't use ARC because when I started evaluating multithreading options (July
2019) and implementing Weave (November 2019) it was not ready at all.
Now at the moment, I'm ready to run my batteries of tests on arc:
[https://github.com/mratsim/weave/blob/33a446ca/weave.nimble#L99-L167](https://github.com/mratsim/weave/blob/33a446ca/weave.nimble#L99-L167)
But I need to be able to at least have custom channels working:
[https://github.com/nim-lang/Nim/issues/13936](https://github.com/nim-lang/Nim/issues/13936)
I do provide already the primitives necessary to have `awaitable` (in the async
sens) Weave jobs via `isReady`, you only need to replace the `sleep()` by
`poll()` here in my own
[waitFor](https://github.com/mratsim/weave/blob/33a446ca4ac6294e664d26693702e3eb1d9af326/weave/datatypes/flowvars.nim#L234-L249):
proc waitFor*[T](p: Pending[T]): T {.inline.} =
## Wait for a pending value
## This blocks the thread until the value is ready
## and then returns it.
preCondition: onSubmitterThread
var backoff = 1
while not p.isReady:
sleep(backoff)
backoff *= 2
if backoff > 16:
backoff = 16
let ok = p.fv.tryComplete(result)
ascertain: ok
cleanup(p.fv)
Run
See writeup/RFC:
[https://github.com/mratsim/weave/issues/132](https://github.com/mratsim/weave/issues/132)
The main issue right now that the GC cannot solve is if we ever want to
implement an `executor` API/concept or at least common `Task` abstraction to
could be dispatch on either:
* Async executors (async dispatch/chronos/libuv)
* Parallel executors (Nim threadpool / Weave)
* An simple executor for blocking tasks like `readline()` because just as you
shouldn't block asyncdispatch, you shouldn't block Weave with non-computational
tasks.
The simple reason why is that tasks in a parallel runtime are threadsafe
closures + metadata, and the GC being thread-local prevents closures from being
sent across threads.
This brings us to why we would want a common Task abstraction
To be honest I'm not sure, but I'm sure we would want a threadsafe async
executor, potentially with task stealing as well, it's not even hard to write
(at least the task stealing side, ~2k lines):
*
[https://github.com/stjepang/async-task](https://github.com/stjepang/async-task)
* [https://github.com/stjepang/smol](https://github.com/stjepang/smol)
but this would require threadsafe closures.
i.e. threadsafe /sendable closures are the core primitive that ARC brings us.
It also gives us `owned` and destructors give use sink which we can use to
enforce ownership when we send tasks across threads. This significantly reduce
the need of Atomic Refcounting schemes as we can just deep-copy the pointers
instead.
For example, we need to enforce ownership of futures. (Hence why I hope that
when a proc is tagged with `sink` parameters it also prevents the caller from
ever reusing that variable again)
Re: How mature is async/threading in Nim?
> Read mratsim's post from the same thread then, > [https://forum.nim-lang.org/t/6352#39200](https://forum.nim-lang.org/t/6352#39200) @mratsim shows no examples of what ARC can do, but I assume that ARC can help with the third scenario that he describes... > You need a shared state, for example a shared hash table. Now you have a > problem :P, if it has infinite lifetime, you can allocate it on the heap and > pass a pointer to it, if not you need to implement atomic refcounting which > is not too difficult with destructors, see for example my refcounted events > in Weave You will note that I wrote "The fact is that I have so far seen no evidence that --gc:arc makes mixing concurrency and parallelism in Nim easier". Yes, ARC does offer a shared heap and makes sharing memory easier, but the interoperability still isn't there between concurrency and parallelism. A shared heap does not solve this. As a side note, I would like to see an example of that third scenario implemented using ARC, why hasn't @mratsim used it by default? :)
Re: How mature is async/threading in Nim?
Adding to that [https://github.com/nim-lang/Nim/issues/14429](https://github.com/nim-lang/Nim/issues/14429) :)
Re: Write Nim in Matlab/Julia style using macros while still deploy to Cloud/PC/GPU/embedded?
Note: the docgen for API is still not ideal, there are internal stuff listed > nd-arrays (vectors and matrices) of integer, floating number, or complex > values. Yes > Slicing, concatenation, transposing... these sorts of array operations. * [https://mratsim.github.io/Arraymancer/tuto.shapeshifting.html](https://mratsim.github.io/Arraymancer/tuto.shapeshifting.html) * CPU: [https://mratsim.github.io/Arraymancer/shapeshifting.html](https://mratsim.github.io/Arraymancer/shapeshifting.html) * CUDA: [https://mratsim.github.io/Arraymancer/shapeshifting_cuda.html](https://mratsim.github.io/Arraymancer/shapeshifting_cuda.html) > Linear algebra (e.g. matrix multiplication, solving linear equations). Matrix multiplication * CPU: [https://mratsim.github.io/Arraymancer/operators_blas_l2l3.html](https://mratsim.github.io/Arraymancer/operators_blas_l2l3.html) * CUDA: [https://mratsim.github.io/Arraymancer/operators_blas_l2l3_cuda.html](https://mratsim.github.io/Arraymancer/operators_blas_l2l3_cuda.html) * OpenCL: [https://mratsim.github.io/Arraymancer/operators_blas_l2l3_opencl.html](https://mratsim.github.io/Arraymancer/operators_blas_l2l3_opencl.html) Solvers, matrix decomposition, PCA ..., CPU only at the moment * [https://mratsim.github.io/Arraymancer/least_squares.html](https://mratsim.github.io/Arraymancer/least_squares.html) * [https://mratsim.github.io/Arraymancer/linear_systems.html](https://mratsim.github.io/Arraymancer/linear_systems.html) * [https://mratsim.github.io/Arraymancer/decomposition.html](https://mratsim.github.io/Arraymancer/decomposition.html) * [https://mratsim.github.io/Arraymancer/pca.html](https://mratsim.github.io/Arraymancer/pca.html) * [https://mratsim.github.io/Arraymancer/decomposition_rand.html](https://mratsim.github.io/Arraymancer/decomposition_rand.html) > 1D FFT, IFFT Not implemented, wrapping MKL FFT can be a weekend project with c2nim or nimterop [https://software.intel.com/content/www/us/en/develop/documentation/mkl-developer-reference-c/top/appendix-e-code-examples/fourier-transform-functions-code-examples/fft-code-examples.html](https://software.intel.com/content/www/us/en/develop/documentation/mkl-developer-reference-c/top/appendix-e-code-examples/fourier-transform-functions-code-examples/fft-code-examples.html) Implemening a pure Nim FFT is something I want to do at one point but lack of time. > All above, running in CPU only (with MKL and/or automated multi-threading > e.g. for large FFT/IFFT) You can use OpenBLAS or MKL in both Neo or Arraymancer. That said, you can write pure Nim code that has performance similar to both OpenBLAS and MKL. I track benchmarks of pure Nim implementation with threading via Laser (using the Nim OpenMP operators) and Weave here: [https://github.com/mratsim/weave/tree/master/benchmarks/matmul_gemm_blas](https://github.com/mratsim/weave/tree/master/benchmarks/matmul_gemm_blas) iterator `||`[S, T](a: S; b: T; annotation: static string = "parallel for"): T ## See https://nim-lang.org/docs/system.html#%7C%7C.i%2CS%2CT%2Cstring iterator `||`[S, T](a: S; b: T; step: Positive; annotation: static string = "parallel for"): T Run Last time I optimized this, I could reach 2.8 TFlops with Weave, 2.8 TFlops with Laser + OpenMP, 2.7 TFlops on OpenMP, 3 TFlops for MKL and 3.1 TFlops with Intel oneDNN ([https://github.com/mratsim/weave/pull/94#issuecomment-571751545](https://github.com/mratsim/weave/pull/94#issuecomment-571751545)) but i started from single-threaded performance of 160GFlops vs Intel and OpenBLAS 200GFlops on a 18-core machine. > GPU Yes but minimal, Cuda and OpenCL at the moment > Statistical functions PCA and SVD are well developped and actually 2x to 10x faster than in any other language (including Sklearn latest optimizations and Facebook's PCA) * [https://github.com/mratsim/Arraymancer/pull/384#issuecomment-536682906](https://github.com/mratsim/Arraymancer/pull/384#issuecomment-536682906) > SPline, Numerical integration and ODE * [https://github.com/HugoGranstrom/numericalnim](https://github.com/HugoGranstrom/numericalnim)
Re: Write Nim in Matlab/Julia style using macros while still deploy to Cloud/PC/GPU/embedded?
I am not sure about the current status of Arraymancer, but I believe it supports most of your requests. Neo, instead, supports only: * Slicing, concatenation, transposing... these sorts of array operations. * Broadcast basic math functions to any array. * Linear algebra (e.g. matrix multiplication, solving linear equations). * All above, running in CPU only (with MKL and/or automated multi-threading e.g. for large FFT/IFFT); plus running in GPU with minimal code change (well, some Lapack things are CUP only for now) The fact is that when I was developing Neo, @mratsim started Arraymancer and improved it a lot. Nowadays, Arraymancer is more advanced and fast: @mratsim has implemented Laser, Weave for multithreading and more, and I don't really see a point in adding many new features to Neo. It will surely be maintained, and what it does, it does decently, but I don't want to duplicate the great effort of @mratsim.
