Re: Current status of Nim for Webassembly?
> It seems to work... Thanks for the link. Then maybe my question is: Under which circumstances would the Nim runtime not run at the root of the call stack? I still don't clearly see the patterns where Nim's GC can / cannot work. Does the `EMSCRIPTEN_KEEPALIVE` play a role in this? What about taking the no-emscripten path as suggested by @arnetheduck and @yglukhov [here](https://forum.nim-lang.org/t/4779) \-- does the emscripten magic have an impact on the GC question?
Re: CSources are gone - How to bootstrap?
Ah, thanks for clarifying -- not it works for me as well ;).
Re: CSources are gone - How to bootstrap?
That's the same `build_all.sh` does, isn't it? I just tried it again with the official `build_all.sh`: git clone https://github.com/nim-lang/Nim.git cd Nim git checkout master sh build_all.sh Run still fails for me when building koch: bin/nim c koch Hint: used config file '/tmp/Nim/config/nim.cfg' [Conf] Hint: used config file 'koch.nim.cfg' [Conf] Hint: system [Processing] /tmp/Nim/lib/system.nim(207, 11) Error: undeclared identifier: 'is' Run
Current status of Nim for Webassembly?
There have been a few webassembly threads already, but it's a bit unclear to me what's the current status of targeting webassembly. For instance [this thread](https://forum.nim-lang.org/t/4399#27529) mentions: > Nims garbage collector only works if the Nim runtime controls the root of the > call stack, if I understand the experts correctly. That's normally not the > case in the browser, but the fact that Nims GC is thread-local might save us > here... How exactly can we setup a project so that "running at the root of the call stack" is accomplished? Does compiling via emscripten and loading the resulting JS glue code satisfy this criterion or not?
Re: CSources are gone - How to bootstrap?
> The idea is that all the existing scripts that happen to not hardcode a > csources commit (which includes all of our own scripts...) continue to work > for good. Do they work? My weekly CI's are failing for ~ half a year because Nim cannot be bootstrapped on the master branch from csources: [https://travis-ci.org/bluenote10/NimData/builds](https://travis-ci.org/bluenote10/NimData/builds)
Re: Generic methods are deprecated?
> AFAIK removal of generic methods is so that under-the-hood we can switch the > methods implementation to a vtable But shouldn't a vtable be able to handle generics that are attached to the base just fine? For instance, the `Test` class has a vtable with a single slot for `call`. An instantiation `Test[int]` would point to an `int` instantiation of `call`, and a `Test[float]` would point to the `float` instantiation. To my knowledge that's how it works in C++. On the other hand, generics not attached to the base, can result in multiple instantiations for a single method. This requires full program analysis, and doesn't play well with vtables.
Re: Best way to represent AST
Or if you prefer inheritance, you can extend the inheritance tree: type Node = ref object of RootObj LitOrOp = ref object of Node Number = object of LitOrOp value: int BinOp = object of LitOrOp left: LitOrOp op: string right: LitOrOp Run
Generic methods are deprecated?
I just noticed that something as simple as
type
Test[T] = ref object of RootObj
method call[T](t: Test[T]) {.base.} = discard
Run
now generates a warning `generic methods are deprecated`.
All my relevant projects rely on generics with methods. Is there a plan for the
future for such projects?
Re: Can I avoid covariance of typedesc arguments?
I'm stupid, I totally forgot that there is the `T is A and A is T` trick that allows for exact type equality comparison. I still think it is a pity that the corresponding PR has been declined ([https://github.com/nim-lang/Nim/pull/8612)](https://github.com/nim-lang/Nim/pull/8612\)). @Stefan_Salewski: The actual background is somewhat involved. In the inheritance library I'm working on, there is a system to automatically perform super/base/parent class initialization. If there is an inheritance chain A => B => C, I have to make sure that when C tries to call its parent constructor, it doesn't accidentally call the constructor of A. This would be possible due to covariance, and result in a bug because B isn't initialized. I'll probably just go for named constructors because it simplifies the problem. I was asking mainly to see what alternatives others use, and the error overload is indeed a nice solution in general.
Can I avoid covariance of typedesc arguments?
Apparently typedesc arguments are covariant. For example: import typetraits type A = object of RootObj B = object of A proc f(T: typedesc[A]) = echo "should only accept A, but not B" echo "T is A: ", T is A echo "name(T): ", name(T) f(A) f(B) # how to prevent this to compile? Run This compiles and prints: should only accept A, but not B T is A: true name(T): A should only accept A, but not B T is A: true name(T): B Run Is there an elegant solution so that I cannot accidentally invoke `f(B)`? The ideal solution (in my particular problem) would be to avoid the function match entirely. Otherwise some when + error statements might be needed, but `when T is A` doesn't work because it also returns true for subtypes, `when name(T) == name(A)` feels hacky.
Re: typed and untyped
On input/argument side, the semantics are clear: * `untyped` passes in the raw AST without semchecking (no type information available) * `typed` performs symbol lookup before passing the argument (type information available) On output side, I'm also confused. I usually only use `untyped` when the macro/template returns an arbitrary AST, and concrete types if it returns a typed expression. The semantics of a `typed` return values has also been discussed here: [https://forum.nim-lang.org/t/4037](https://forum.nim-lang.org/t/4037)
Re: Side effects of typed macro arguments?
Okay, thanks for the clarification. Than I think the most natural thing is to use blocks to get a limited scope.
Re: Side effects of typed macro arguments?
Maybe I shouldn't have obfuscated the example by the two macro calls. Of course you're right: The first macro simply gets expanded to the typed macro call. The actual use case is somewhat involved, but maybe not super relevant. I was mainly puzzled by the behavior, and I'm still wondering if it is a bug or intentional. Note that because typed argument actually affect the callsite you can do things like this: macro takesTwo(a, b: typed): untyped = result = newStmtList() takesTwo() do : let x = 1 do: x # but don't dare to redefine x here ;) Run where one typed argument gets its type information from the evaluation of another argument. I'm wondering how much Nim code relies on that.
Side effects of typed macro arguments?
Is it expected behavior that typed macro arguments have a side effect on call site? For example I was surprised that this code compiles: macro typedMacro(body: typed): untyped = result = newCall(ident "echo", newStrLitNode "Hello World") macro untypedMacro(body: untyped): untyped = result = newCall(ident "typedMacro", body) untypedMacro(): echo "here" proc test(x: int) = echo x test(1) Run My assumption was: I'm passing an untyped block into a macro and locally I just get the resulting AST `echo "Hello World"`. I didn't expect that `test` is defined afterwards, because it is not part of the final AST. Is that by design? What is the best way to get side-effects-free typed arguments? Should I simply wrap them in a block?
Re: Problem with export marker and macros
I was also looking for some postfix operator syntax in a DSL, and I noticed that `*` Run (how to write that in as inline RST?) can actually be used. Looks like a postfix operator, but semantically it is just an `AccQuoted` argument to the preceding `Command` node. In your case: activations: act sigmoid`*`: fn: 1 / (1 + exp(-x)) deriv: fx * (1 - fx) act tanh`*`: fn: tanh(x) deriv: (let t = tanh(fx); 1 - t * t) Run The relevant part of the AST is: Command Ident "act" Command Ident "sigmoid" AccQuoted Ident "*" Run Slightly non-standard, but works well in my purpose and looks even better in an editor with special syntax highlighting for acc-quoted ops.
Re: How to immutably initialize parent object with private fields?
@Araq: Great to hear that there is some activity regarding write tracking, my most anticipated Nim feature ;). I think this is an independent problem though, that rather relates to object construction / initialization, not really related to immutability (my title was misleading). I'm not sure if it is worthy of a RFC, but I tried to sum it up more clearly here: [https://github.com/nim-lang/RFCs/issues/145](https://github.com/nim-lang/RFCs/issues/145) It could very well be that I'm missing something ;).
Re: closure_methods: OOP purely based on closures
I didn't know that this can be solved by forwarding from one macro to another, will try that, looks really nice. I will experiment a bit more regarding the field accessors when I'm back from vacation. Overall this has a bit lower priority to me, because the use cases I have barely require plain getter/setter. If a situation requires exposing raw data, plain objects are a perfectly fine solution already. I'd use this closure based approach exactly when I want to hide raw data behind an abstract interface.
Re: closure_methods: OOP purely based on closures
@gemath That's also something I had in mind ;). The problem with autogenerating getters/setters for `var x*` expressions is that closures don't allow the regular `field` / `field=` syntax as far as I can see. For instance: type Obj = ref object step: proc(): int `step=`: proc(x: int) let o = Obj(step: proc(): int = 0, `step=`: proc(x: int) = echo x) echo o.step() o.`step=`(1) # I guess only this syntax is possible Run So you cannot just use something nice like `o.step = 1`. Thus, it's probably better to use different names for the getter and setter. The auto-generated getter/setter could follow a fixed convention like `step`/`setStep` or `getStep`/`setStep`. In the end I preferred to give the user control over how they call them with the current getter/setter shortcuts. I also tried to use the `Counter of SomeBaseClass` syntax, but for the overload analysis it is important to pass in the base class as a `typed` macro argument. `Counter of SomeBaseClass` would require an `untyped` argument, and I don't think it would be possible to access the type impl in this case. That's why I went for the slightly less appealing `classOf(Sub, Base)` syntax, and that's also why generic subclasses will probably require a minor repetition `classOfGeneric(Sub[X, Y], Base, Base[X])` \-- the first appearance of `Base` is a typed arg, the one with the generics is untyped.
Re: closure_methods: OOP purely based on closures
So far I only ran some stupid benchmarks on dispatch speed [here](https://github.com/bluenote10/closure_methods/blob/master/benchmarks/basic_dispatch_closures.nim). and in that terms using closures is a bit faster than the branch based dispatch of methods, even if the number of subclasses is still low (but nothing substantial). Obviously using closures is the wrong tool if a problem requires raw field access in tight loops (e.g. vector math library). The design rather aims at use cases that are well suited for OOP abstraction, for instance a React-like web components library, where field access is negligible. Do you have a good idea of a benchmark that neither focuses too much on field access nor on the O(1) vs O(N) dispatch difference?
Re: Owned refs
I really like the proposal, but I may not fully understand the core idea ;). What exactly do you mean by: > In a release build the RC operations can be left out and with a type based > allocator we still have memory safety! Does this refer to memory safety as in "no memory leaks" or as in "no memory corruption"? I always thought it is the latter. What would happen for instance in this example: var n: owned Node = fromSomewhere() let dangling = unowned(n) if random() > 1e5: # imagine lots of code here dangling = nil else: # imagine lots of code here # but programmer forgets to remove the unowned ref # reassignment => free old owned Node n = Node(data: 4) if not dangling.isNil(): echo dangling.data Run In a debug build, the reassignment would lead to an abort if we ever execute the unlikely branch that forgets to remove the ref. But in a release build the last line would read corrupted memory, wouldn't it? Just because it points to memory of the right type, it doesn't mean that it is not an issue?
closure_methods: OOP purely based on closures
Yet another OOP macro ;). [*] I was wondering if it would be possible to write classes in Nim entirely by using closures. A good opportunity to work on my meta programming skills. At first the approach looked a bit weird, but now I feel that the resulting DSL might actually work quite well. What do you think? [closure_methods](https://github.com/bluenote10/closure_methods) [*] Actually I was searching for OOP macros on Nimble but couldn't find much...
Re: How to immutably initialize parent object with private fields?
@Rania_d Note that the issue you linked is only a problem in languages that don't allow named parameters. It's not a problem in Nim because it has named parameters, so no need for the builder pattern. Nim's problem is rather that it doesn't allow to combine immutable objects with inheritance out of the box. @gemath Good idea, so it comes down to either setting the additional fields of the parent object vs the fields of the derived object.
Re: How to immutably initialize parent object with private fields?
> The accessor are only visible for the internal library proc but not exported
> outside of the library.
In this case I cannot rely on hiding the fields via re-exporting, because the
library is by design supposed to be extended on user side. Think of an UI
component library, where users should be able to define their own components.
@lscrd Good points about the runtime checks, but yes compile time security
would be preferred.
I have now found two solutions that seem to satisfy the requirements:
1\. Macro solution:
# on lib side
template privateInitializer(element: typed, idString: string): untyped =
element.id = idString
element
macro newElement*(T: typedesc, id: string, args: varargs[untyped]): untyped
=
# convert varargs into an object constructor call of T
let constructor = newTree(nnkObjConstr)
constructor.add(T.getTypeInst[1])
for arg in args:
expectKind(arg, nnkExprEqExpr)
constructor.add(newColonExpr(arg[0], arg[1]))
# apply post construction initialization of parent fields
result = newCall(bindSym"privateInitializer", constructor, id)
# on user side
newElement(ElementA, id="A", a=1)
Run
2\. Template solution with an explicit check that only object constructors are
passed in:
# on lib side
macro verifyObjectConstructor(x: typed): untyped =
if x.kind != nnkObjConstr:
error($x.repr[0..^1] & " is not an object constructor")
template newElement*(idString: string, constructor: typed): untyped =
verifyObjectConstructor(constructor)
let element = constructor
element.id = idString
element
# on user side
newElement("A", ElementA(a: 1))
# abusing the template as an arbitrary setter is now a compile time error:
let el = ElementA(a: 1)
newElement("A", el)
Run
@Araq: What is your opinion on this? I'm wondering if it would make sense to
differentiate visibility for construction vs field access. As far as I can see
a lot of visibility problems would be solved if there was an intermediate
between fully-hidden and fully-exposed which is "exposed in object
construction". Would it for instance make sense to have
type
Element* = ref object of RootObj
id {.initializable.}: string
Run
so that public access to `id` is prevented in general, but `id` can be passed
into the constructor including subtype constructors?
Re: How to immutably initialize parent object with private fields?
@lscrd: The `initElement` is also a setter despite its name ;). You cannot stop
users from calling `element.initElement("different id")` and mutating the field
at any point in time, which is what I'm trying to avoid.
In fact that is also true for my template constructor idea via
`newElement("different id", existingElement)`. If anything the template makes
it a bit less obvious. Maybe what I really need is a macro taking a
typedesc/generic and a list of constructor args that are forwarded to the
subtype construction. Then the macro can really only be used for constructing
new elements with injected `id` modifications.
Re: How to immutably initialize parent object with private fields?
@Stefan_Salewski: That what I meant with mutable setters ;). I'm trying to
avoid them so that user can cannot arbitrarily reset the field.
I finally had an idea that I can at least abstract away the mutation of the
field:
# in lib.nim
template newElement*(idString: string, body: typed): untyped =
let element = body
element.id = idString
element
Run
On user side construction now works like this:
type
ElementA = ref object of Element
a: int
ElementB = ref object of Element
b: int
proc newElementA(): ElementA =
newElement("A", ElementA(a: 1))
proc newElementB(): ElementB =
newElement("B", ElementB(b: 2))
Run
Not sure if there is something nicer, but this doesn't look too bad to me...
How to immutably initialize parent object with private fields?
I'm running into this problem in every project of mine, and I simply can't find
a good solution, although the problem seems very basic.
Let's say I write a dummy `lib.nim` like this:
# The lib provides a definition of an abstract base type
# without any actual instantiations...
type
Element* = ref object of RootObj
# Example for a field that should be an implementation
# detail, used only internally in `lib`. It should
# neither be public nor should there be getter/setter
# so that we can rely on the fact that it does not
# change after construction.
id: string
proc run*(elements: openarray[Element]) =
for element in elements:
echo "Touching element: " & element.id
Run
The library is intended to be instantiated on user side, for example:
import lib
type
ElementA = ref object of Element
ElementB = ref object of Element
# But how to construct them?
proc newElementA(): ElementA =
ElementA(id: "A")
proc newElementB(): ElementB =
ElementB(id: "B")
let elements = @[
newElementA().Element,
newElementB(),
]
run(elements)
Run
This doesn't compile because the user has no access to `id` at all, even in the
constructor. Usually I provide a constructing proc in the scope of the type,
e.g.:
# in lib.nim
proc newElement*(id: string): Element =
Element(id: id)
Run
Now users can construct an `Element`, but I'm not sure if it helps constructing
subtypes. This crashes at runtime with an invalid conversion (kind of expected):
proc newElementA(): ElementA =
result = newElement("A").ElementA
Run
I also don't want to add setter procs/templates to `lib.nim` because than the
field gets fully exposed.
Am I missing some basic template/macro tricks to solve this basic problem? I
have spend hours going back and forth making fields private and public again or
moving code around because I can't seem to get the visibility right.
Re: watchcode: not written in Nim, but at least for Nim ;)
No particular challenge, really. In the end I went for the Python solution because (1) the problem is too trivial to require a serious language, (2) the result is arguably more accessible, (3) was a bit faster to write because the file monitoring libraries in Nim aren't quite as mature. I just wanted to have it done quickly to use it for other Nim projects.
watchcode: not written in Nim, but at least for Nim ;)
I have just published a small project called [watchcode](https://github.com/bluenote10/watchcode). As an exception, it's not a Nim project, but it was written 100% with Nim in mind ;). I was never fully satisfied with other solutions for code watching/rerunning, so I had to give it a go and I'm quite happy with the result. Maybe it is useful for other Nim developers as well.
Re: Should we get rid of style insensitivity?
I would have been interested in a vote, but apparently I'm the minority ;). Personally I find the case insensitivity very off-putting and it was one of the reasons why I stayed away from Nim the first times I came across it. > I appreciate when Nim forgives me a typo That's my main concern about it. I like consistency, and I don't want inconsistent naming in my code base. I want the compiler to help me get it right exactly. I never saw it as a feature, rather a quirk that I can live with. Yet, sometimes I still feel like I'm manually doing the work of the compiler when I'm double checking that I use consistent style. And I get distracted when reading foreign Nim code that switches between equivalent names for no apparent reason, especially when done in a local scope, where the syntax becomes obviously inferior to a consistent naming.
Re: Are there examples of the 'Elm architecture' for Nim?
I had also written some small demos here: [https://github.com/bluenote10/KaraxExamples](https://github.com/bluenote10/KaraxExamples)
Re: TechEmpower Web Framework Benchmarks: Round 10
Doesn't the compiler pick up this file? [https://github.com/tbrand/which_is_the_fastest/blob/master/nim/jester/server_nim_jester.nim.cfg](https://github.com/tbrand/which_is_the_fastest/blob/master/nim/jester/server_nim_jester.nim.cfg)
Re: Introducing loopfusion: loop over any number of sequences of any single type.
Looks interesting, but why does the macro generate a for loop plus a zip iterator (which itself contains another for loop) instead of just a single for loop with injected symbols? forEachIndexed j, [x, y, z], [a, b, c]: body # would expand to: for j in 0 ..< commonLengthWithSanityCheck(a, b, c): let x = a[j] let y = b[j] let z = c[j] body Note: The length of `b` in your example differs. I was wondering if this is not handled, but I think it is, so probably just a typo . Syntactically I would prefer something that aligns the loop variables with the containers, i.e., forEachIndexed j, x in xs, y in ys, z in zs: ...
Re: constness of refs & ptrs
I would also be interested in a bit more information on writetracking, it seems like a perfect solution to the constness problem. How can it be enabled and what is the current status/plans? > At this point we don't need more features that don't pull in their weight. That's surprising. I think solving the ref constness problem for good would have a huge benefit, it always felt like a big missing piece in Nim. I was hoping this would make it into version 1, especially because it sounded like writetracking is the foundation for move semantics etc.
Re: block expression
There should be no need for such a template, this must be a regression. Since [this PR](https://github.com/nim-lang/Nim/pull/6695/files) blocks should be an expression (see [corresponding issue](https://github.com/nim-lang/Nim/issues/3827)). There is also a test case. However the test case indeed seems to fail now on devel, not sure if testament somehow ignores the test...
Re: Macros: How to parse a string and get the type of the expression?
Ah, I think I found a solution:
import macros, strutils
macro typecheckedAst(x: typed): untyped =
echo "calling typecheck on expression: ", x.repr
echo "with type: ", x.getType().treeRepr()
# We can build a type specific AST here:
let typ = x.getType()
if sameType(typ, bindSym"int"):
echo "it's an int"
result = newStrLitNode("AST for int")
elif sameType(typ, bindSym"string"):
echo "it's a string"
result = newStrLitNode("AST for string")
else:
echo "unknown type"
echo " => resulting type dependent AST: ", result.repr
macro fmt*(fmt: static[string]): untyped =
let fields = fmt.split(":")
assert fields.len == 2
let lhs = fields[0]
let rhs = fields[1]
# We probably can't get the type of `e` here, but we can delegate
# all type specific AST generation into the `typecheckedAst` macro
let e = parseExpr(lhs)
result = newCall(ident"typecheckedAst", e)
block:
let x = 1
echo fmt"x + x:"
block:
let x = "test"
echo fmt"x & x:"
So the trick is to use `parseExpr` in the outer macro and delegate the type
specific AST generation into a sub-macro which takes a `typed` expression.
Re: Macros: How to parse a string and get the type of the expression?
@planhths: What I'm looking for is the slightly more advanced version of that, i.e., a macro that generates a different AST depending on the type of that `parseExpr(v)` in your example.
Re: Macros: How to parse a string and get the type of the expression?
Sorry for bumping an old thread, but it's for a good reason (the [string
interpolation PR](https://github.com/nim-lang/Nim/pull/6507)). I'm still
struggling to achieve the same from within a macro:
macro parse(x: string): untyped =
echo "calling parse on: ", x
result = parseExpr(x.strVal)
macro typecheck(x: typed): untyped =
echo "calling typecheck on: ", x.getType().repr()
result = newStmtList()
template stringParseTest(x: string) = typecheck parse x
macro fmt*(fmt: static[string]): untyped =
let fields = fmt.split(":")
assert fields.len == 2
let lhs = fields[0]
let rhs = fields[1]
echo "Trying to get type of: ", lhs
# Neither of the following works, because the ``x.strVal`` produces:
# Error: field 'strVal' cannot be found
# Also, it is strange that the value of x within parse is "lhs" and not
# its actual value.
when false:
let lhsType = stringParseTest(lhs)
when false:
let lhsType = typecheck(parse(lhs))
when false: # in general calling parse doesn't seems to work due to the
value issue:
parse(lhs)
# So it looks like I have to call parseExpr directly here. This gives the
expected
# result for the expression, but calling typecheck on it does not work,
because of:
# Error: expression '' has no type (or is ambiguous)
when false:
let e = parseExpr(lhs)
echo e.treeRepr
let lhsType = typecheck(e)
# Similarly any attempt to call getType() directly on the expressions
fails with
# node has no type.
when false:
let e = parseExpr(lhs)
echo e.treeRepr
let lhsType = e.getType()
block:
let x = 1
fmt"x + x:"
block:
let x = "test"
fmt"x & x:"
I feel like I'm just missing the right kind of nesting. Or maybe it isn't
possible when the outer-most thing is a macro, and I really need to wrap
everything into another outer template?
Re: floating point output formating
@jzakiya: Do you think that is helpful: [https://github.com/nim-lang/Nim/pull/6704](https://github.com/nim-lang/Nim/pull/6704)
Re: floating point output formating
@Udiknedormin I strongly think it should be part of the language. Developers fear dependencies[*], so people will continue to use workarounds instead of using string interpolation via an external dependency. Also, the Nim core itself could benefit from it in some places ([random example](https://github.com/nim-lang/Nim/blob/87a46765eee6601ccc53b2c38d1c8f3a9b045124/lib/system/nimscript.nim#L285) or every line with more ~5+ concatenating `&` and the notorious opening and closing `"`). Sometimes I don't embed helpful information in my exception strings, because typing out the manual concatenation is tedious (and I don't want to add a dependence for just 1 line of code), whereas in e.g. Scala I would, because it is so easy (`f"Error: blah blah because x = $x and y = $y while running in mode '$mode'"` if I counted correctly that would take 6 `&` and 8 `"` in Nim). It's a general pattern that in languages where string interpolation is included, developers tend to produce higher quality output strings on average compared to languages where there is a larger barrier for achieving the same. [*] I'm an author of one of the string interpolation libs out there, but I've never dared to use it in my other libraries because of dependency avoidance
Re: floating point output formating
import strutils let num = 10.123456789 echo num.formatFloat(ffDecimal, 4)
Re: floating point output formating
And hopefully we will soon have string interpolation in Nim ([PR](https://github.com/nim-lang/Nim/pull/6507)). Then it would be something like this (or similar, language is still in discussion): echo "$x%.4f"
Re: Has anyone considered if Nim would be a good fit for a "serverless" architecture?
I was experimenting with distributed computing a bit. The idea is slightly different, because the same binary has to be published to each cluster node and it can run in different roles client/master/worker. It's far from being usable, but at least it can already publish data into worker memory and run arbitrary code remotely on a worker. Usage looks like that: [https://github.com/bluenote10/Hydra/blob/master/testapp1.nim](https://github.com/bluenote10/Hydra/blob/master/testapp1.nim)
Re: Cannot get name of type using typedesc
I think your third example should work if you `import typetraits`. Actually let's test the forum feature... import typetraits proc test3*(T: typedesc) = echo(T.name) test3(int)
Re: DSL for generating/animating SVG/GIF
Good point, I didn't think of the JS backend so far.
Re: Introducing Reel Valley
> @bluenote, thank you! I wonder based on what you made that quality assumption > though The high quality media? I assume what I saw was partly in-game graphics.
Re: Introducing Reel Valley
I'm not on Facebook, so I can't really judge, but from what I could see this looks like super high quality, really impressive.
DSL for generating/animating SVG/GIF
Just to let you know: The next time you have to work on some graphics/illustrations/animations, you can do so in Nim . [https://github.com/bluenote10/NimSvg](https://github.com/bluenote10/NimSvg)
Re: FSMonitor
I wish we could go down the other route and make `fsmonitor` cross-platform by abstracting away the inotify stuff. The reason why having it in the standard library could be beneficial: One of the things I'm missing the most compared to Scala development is SBT's `~compile`/`~run`/`~test` functionality, which allows to easily set-up source code watchers and re-run the compiler (and optionally arbitrary entry points or tests) whenever the source code changes. If the functionality gets pushed into third party libraries it will be even more difficult to integrate such functionality by default into Nim's build approaches. I currently always fall back to hacky/non-cross-platform bash+inotify or even nodejs based watchers, and it would be much nicer if source code watching could become standard nimscript functionality. I have it on my todo list to look into that, but I'm not sure when I can get to work on it. Wasn't there a thread asking for some beginner challenges, maybe that would work?
Re: optional int parameters for a proc
Actually there's no need for external libs or rolling your own solution. The standard library has an `Option[T]` and is exactly suited for this.
What assumption can be made on pointers returned by getTypeInfo?
Given two types A and B, is it safe to assume that the following holds over the entire runtime of a program: Type A == Type B if and only if the pointers returned by `getTypeInfo` are the same? Or is it possible that e.g. the pointer `getTypeInfo(anInt)` can point to different addresses over the course of execution. Background: I need to maintain a hash table, where the keys are types. If it is safe to make this assumption the addresses themselves could be used as a hash instead of actually comparing the underlying `PNimTypes` (implementing such a types comparison looks tricky on first glance, or is this already solved somewhere within Nim)?
Re: nim-random not that random at all?
Are you on Windows? My guess is that this is mainly an issue of `randomize` and the low resolution of the timer used for initialization [here](https://github.com/nim-lang/Nim/blob/47a7f2fff795be3c4c28059cfa573f5d6069af44/lib/pure/random.nim#L126).
Re: Caller line numbers in templates?
I'm just trying to figure out exactly the same. Did you find a solution to that?
Re: Arraymancer - A n-dimensional array / tensor library
A late reply because I was hoping to dive into this a bit deeper before replying. But due to lack of time, a high-level feedback must suffice: This looks awesome! I completely agree with your observation that there is a gap between developing prototypes e.g. in Python and bringing them into production -- not only in deep learning, but data science in general. And I also think that Nim's feature set would be perfect to fill this gap. A quick question on using statically-typed tensors: I assume that this implies that the topolgy of a network cannot be dynamic at all? I'm wondering if there are good work-arounds to situations where dynamic network topologies are required, for instance when a model wants to choose its number of hidden layer nodes iteratively, picking the best model variant. Are dynamically typed tensors an option or would that defeat the design / performance?
Re: Import from parent directories
I'm not sure if I understand the question correctly. An example project structure: submodA/submodA.nim submodB/submodB.nim parent.nim Now you can do in `submodA.nim` for instance: import ../parent import ../submodB/submodB Maybe your file names require to put the paths in quotes?
Re: How to transform compile time into runtime data
Thanks guys, good suggestions, that should do the job.
How to transform compile time into runtime data
I'm working on exactly the same problem as @andrea did a [while
ago](https://forum.nim-lang.org/t/1560) (registering procs at compile-time) and
I'm running into the same difficulty: How can I capture the content of a
compile time variable at the end of the entire compilation and turn it into a
runtime variable.
Imagine there is a `remote.nim` module (omitting ids & mapping):
import macros
var registeredProcs {.compileTime.} = newSeq[string]()
macro remote*(n: untyped): untyped =
let procName = n[0]
registeredProcs &= $procName
echo registeredProcs
result = n
proc square(x: float): float {.remote.} = x * x
proc cubic(x: float): float {.remote.} = x * x * x
# this compiles, but it doesn't really work
const procs = registeredProcs
proc genericCall*(id: string, x: float): float =
echo "looking up ", id, " in ", procs
for funcId in procs:
if id == funcId:
echo "would call id: ", funcId
# locally in this module it works fine
discard genericCall("square", 1)
discard genericCall("cubic", 1)
The problem is that external usage doesn't seem to work. I.e., if there is a
`user.nim`:
import remote as remote_module
proc userSquare(x: float): float {.remote.} = x * x
proc userCubic(x: float): float {.remote.} = x * x * x
discard genericCall("userSquare", 1)
discard genericCall("userCubic", 1)
I can see that the usages of `{.remote.}` on client side do modify the
`registeredProcs`, but I can't seem to capture them as well in the runtime
variable `procs`. Any ideas how to approach this? I feel like I need to run
some static code after everything else is compiled, but that is probably
impossible.
Re: Add Nim to The Computer Language Benchmarks Game?
I was a bit annoyed that they ban certain languages from their "game", so I started to collect my own set of silly benchmarks (plus a framework to make defining, running, and visualizing benchmarks easy): [https://bluenote10.github.io/SimpleLanguageBenchmarks](https://bluenote10.github.io/SimpleLanguageBenchmarks)/ The goal was to have a different focus by benchmarking simple/idiomatic implementations, e.g. to find out what performance I can expect from standard libraries in different languages. The Benchmark Game mainly compares what you can achieve over years of optimization => not the performance you get when you start out in a language. Unfortunately, I didn't get too far with the project, but Nim already looks really good . If anyone is interested in adding more benchmarks/implementation we could just run our ban-free benchmark suite.
Re: Do notation without parentheses
Most likely this is not really intended. For instance, this now compiles, but produces invalid C code: import future proc takesFunc(f: () -> void) = f() takesFunc do: var x = 1 echo x Probably best to track this as an issue: [https://github.com/nim-lang/Nim/issues/5963](https://github.com/nim-lang/Nim/issues/5963)
Re: Linear algebra library
That's looking great. How does the scope of the two compare exactly?
Re: Do notation without parentheses
> The code tries to pass a parameter with the wrong type to that proc, yes. But > the compiler doesn't even get that far: I'm not sure about that. So with having just one body or no conflicts you would expect a type error? But this compiles and works just fine: import future proc takesFunc(f: () -> void) = echo "running f()" f() echo "done running f()" takesFunc do: var x = 1 echo x takesFunc do: var y = 2 echo y Output: running f() 1 done running f() running f() 2 done running f()
Do notation without parentheses
I'm still getting confused about the do notation. In particular I'm surprised that the following gives an error "redefinition of x": import future proc takesFunc(f: () -> void) = f() takesFunc do: var x = 1 takesFunc do: var x = 1 The manual mentions something that `do` without parenthesis is just a block. I don't understand that, what is the compiler passing to the function in this case? Adding some empty parentheses after the `do` solves the problem, but if the version without the parentheses is not creating an anonymous proc, shouldn't this lead to a more meaningful error? Otherwise finding such a bug is a bit by chance like in my case with the redefinition.
Re: Tell Visual Studio Code to use JS backend?
Ignore me: Overloading on `string` and `cstring` does not make much sense here. Problem solved by omitting the `string` version... Would still be interested in the backend config question though.
Tell Visual Studio Code to use JS backend?
I'm currently experimenting with wrapping some JS test framework. A minimal
example:
import future
proc describe(description: cstring, body: () -> void) {.importc.}
proc describe(description: string, body: () -> void) =
describe(description.cstring, body)
proc it(description: cstring, body: () -> void) {.importc.}
proc it(description: string, body: () -> void) =
it(description.cstring, body)
type
Expect = ref object
proc expect[T](x: T): Expect {.importc.}
proc toBe[T](e: Expect, x: T) {.importcpp.}
describe("The test suite"):
it("should work"):
var a = 1
expect(a).toBe(1)
When using the JS backend, this compiles and works pretty well already. However
using the standard `c` backend, this produces an error `Error: internal error:
environment misses: a`. Two questions:
* Should we bother that it doesn't compile with `c`? Known issue? I think
part of the problem is that the `describe` and `it` functions have been
overloaded with both `string` and `cstring`.
* When writing some larger test code, the Visual Studio Code linter flags
this internal error for almost every line of code, which is pretty annoying to
work with. Typically I can use `.nims` or `nim.cfg` to modify the way Visual
Studio Code compiles (or rather "nimsuggests") a file, e.g., for define
switches. I was trying to find a way to configure the backend as well from the
config files, but I haven't found a way yet. Is this possible?
Re: Surprising floating point equality
I would agree that this is not quite as expected. In the generated C code, the 47.11'f32 literal becomes 4.7109e+01, i.e., it has higher precision than `y`. One might think that an explicit conversion should help, but I think it is optimised away. If you need a work-around, wrapping the conversion in a function does work: proc forceF32[T](x: T): float32 = x.float32 let y: float32 = 47.11'f32 # the 'f32 actually doesn't matter assert forceF32(47.11'f32) == y # ok assert float32(47.11'f32) == y # fails
Re: Announcing Karax -- Single page applications for Nim
In fact, it is maybe not even too far away: include karaxprelude import jstrutils, dom, future type Model = ref object toggle: bool EventHandler = (Event, VNode) -> void EventHandlerModel = (Model, Event, VNode) -> void proc curry(handler: EventHandlerModel, model: Model): EventHandler = result = (ev: Event, n: VNode) => handler(model, ev, n) proc onClick(model: Model, ev: Event, n: VNode) = kout(ev) kout(model) model.toggle = model.toggle xor true proc view(model: Model): VNode = result = buildHtml(): tdiv: button(onclick=curry(onClick, model)): text "click me" tdiv: text "Toggle state:" tdiv: if model.toggle: text "true" else: text "false" proc runMain() = var model = Model(toggle: true) proc renderer(): VNode = # var model = Model(toggle: true) # init here does not work view(model) setRenderer renderer runMain() At first I was initializing the `model` in the renderer proc. This had the result that the view never updated -- it's not immediately clear to me why it has to be in the outer scope. Adding an Elm-like message approach should also be possible by constructing some `Msg` in each event handler and calling an `update` function. Okay, in contrast to Elm the model is updated in-place, but maybe it is even beneficial to keep the model update close to how it works natively. The only issue is that every event handler has to be curried. So my question probably becomes: Is it possible to automatize this "injection" of the model into the event handlers, so that an event handler `(Model, Event, VNode) -> void` can be passed directly to `on...`.
Re: Announcing Karax -- Single page applications for Nim
@Araq: Sound very interesting, really looking forward in which direction this will develop. +1 for the Elm inspiration and not following the the one-file-component path. And compared to Elm, the Nim syntax really shines for the templating stuff. I've been wondering for some time, if it would be possible to achieve a design which is even closer to the Elm architecture? On first glance it would be so nice to write stuff like this: type: Model = object # ... Msg = object # typically an object variant proc init(): Model = ... proc update(msg: Msg, model: Model): Model = ... proc view[Msg](model): Html[Msg] = ... karax.runMain(init, update, view) Are there any plans to go in that direction or are there technical obstacles which prevent such a design?
Re: Announcing Karax -- Single page applications for Nim
Indeed, great idea! Did you had a look at Vue for inspiration? Many people consider it the next big thing after React, and in terms of performance it has already [raised the bar](https://vuejs.org/v2/guide/comparison.html#Performance-Profiles) a bit. But performance is typically not the most important aspect for choosing a JS framework. Probably karax would benefit the most by making it more accessible to JS users. Some random questions (not necessarily to be answered here, just points for documentation). * How to interact with third party JS libraries? (A general question that is most likely answered in the JS backend documentation itself; just to point out that it is one of the first questions that come to mind, and linking it from karax would be good.) * How does state handling work. In a Vue/React component, I know pretty much were to look for state, in the examples I find it very hard to see. * Why aren't components split into template/code(/style)? * How do components communicate? * What about the tooling, i.e., how does building / bundling / browser test running / debugging (source maps) work? An idea for the documentation would be to map JS artefacts like npm/webpack/mocha/karma/selenium to possible solutions. * By the way, my first thought when looking at the code was: All cstring here, I'm scared -- wait, it's not even compiling to C? I think I get now why it has to be like that, but getting rid of that c could definitely reduce the scare factor for newbies .
Re: Plans regarding named tuples
Ah, maybe my assumptions were just wrong. All I knew was that something like this is not possible: # can't use `Anonymous` as return type proc test(): Anonymous = type Anonymous = object x: int y: int result = Anonymous(x: 0, y: 0) But it looks like objects can already be anonymous, because this seem to work: proc test(): auto = type Anonymous = object x: int y: int result = Anonymous(x: 0, y: 0) Not being able to write out the type explicitly feels weird though. And things get messy when structural equivalence matters: proc testA(): auto = type Anonymous = object x: int y: int result = Anonymous(x: 0, y: 0) proc testB(): auto = type Anonymous = object x: int y: int result = Anonymous(x: 0, y: 0) # this works echo testA() == testA() # this doesn't echo testA() == testB() So maybe objects aren't too far away to replace named tuples already. But I have to say that I still like the simplicity of named tuples a lot and will surely miss the nice syntax .
Re: Plans regarding named tuples
> Please elaborate.
I'm mainly aware of the situation in Scala, where the lack of named tuples is
the reason why type-safe schema transformation is rather limited. When working
with typed data, there are basically two options:
* Use unnamed tuples, which is not really an option, because it either
hard-codes column positions (=> unreadable) or requires tedious pattern
matching over all fields.
* Use (case) classes, which is the standard solution: You write out a case
class, which involves typing out all the field names/types once. The problem is
that transforming the data cannot be done automatically. Let's assume the input
data has 30 columns, so we have to write a first class RawInput with 30 fields.
In a later processing step we might want to remove a few columns. This again
requires to define a new class ReducedInput with 20+ fields. Eventually we
might want to add a bunch of derived columns, and again we have to introduce a
new type. The problem can be mitigated by inheritance/traits, but it still
remains a work-around which is not very convenient to work with.
In Nim, the same can be solved very elegantly by just transforming/constructing
named tuples everywhere. That's what the DSL looks like:
# A const schema definition is required once. Ideally this is the
# only point where we have to type out our 30 columns.
const schema = ... # array with field information
# For here on, it is just a bunch of macros performing named tuple
transformations
let df = DF.fromText("test.csv")
.map(schemaParser(schema, ";"))
# Projection can use whichever is shorter to type
df.map(t => t.projectAway(fields, to, remove))
df.map(t => t.projectTo(fields, to, keep))
# Adding new fields also does not require repeating existing fields
df.map(t => t.addFields(length: sqrt(t.x^2, t.y^2))
# Eventually even the schema of a join can be computed statically:
let joined = dfA.join(dfB, on=[joinField])
This should also play nicely with structural typing in Nim, e.g., passing data
frames to functions can be done generically, and does not require to write out
field names explicitly.
I'm not sure how this would work with objects. Since they are nominal, I guess
they would have to be made explicitly available in the outer scope. Currently I
leave it up to the user if they want to define their types explicitly, for
instance via this macro:
type
MyRowType = schemaType(schema)
proc myExplictlyTypedProc(df: DataFrame[MyRowType]) = ...
What I wanted to avoid is that a user has to explicitly name their types for
each transformation.
Re: Nim core developer wanted
That's really great news! Do you already know when he will start and what will be the focus?
Re: Designing a data frame API
@andrea: You are right, that is probably not yet very clear from the docs. They
currently rely too much on being familiar with the concept of caching in
frameworks like Spark (in fact, the API is modelled after Spark's
[RDDs](https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD),
but for now without the distributed aspect).
The idea is basically to offer both lazy operations and eager operations. If
your code looks like this
let df = DF.fromFile("data.csv").map(schemaParser, ',')
let averageAge = df.map(x => x.age).mean()
let maxHeight = df.map(x => x.height).max()
let longestName = df.map(x = x.name.len).max()
the operations are completely lazy, i.e., you would read the file three times
from scratch, apply the parser each time, and continue with the respective
operation. You would use this approach when your data is huge and it can't fit
into memory entirely. This is obviously inefficient for small data which would
fit into memory.
If you write the exact same code, but replace the first line by
let df = DF.fromFile("data.csv").map(schemaParser, ',').cache()
the result of the mapping operation would be persisted in memory (using a
`seq[T]` internally). Thus, all following operations will use the cache, and
you would have to read and parse the file only once. Caching gives very good
control over trading off memory usage over recomputation. You can cache the
computation pipeline at any point, as long as you have the required memory.
Typically you would use `cache` after having done some (potentially complex)
computations which are required for multiple consecutive computations like
repeated iteration in machine learning algorithms. In some use cases the data
can also be preprocessed to make it fit into memory, e.g., by filtering to the
interesting bits of the data, downsampling, or simply projecting the input data
down to a single column.
Other differences to purely lazy libraries are that data frames require
operations like sort or groupby, which require some sort of internal
persistence. For now I'm just using an implementation which falls back on
storing the entire data in memory, but I hope that I can add some spill-to-disk
features later.
Re: Designing a data frame API
@chemist69: Thanks! And by the way, I forgot to mention the most important
feature from a practical perspective: There already is a simple browser viewer
available via `df.openInBrowser()`. No plotting yet, but at least handy for
quick data inspection.
@jlp765: Handling different types is already possible, as illustrated in the
readme example as well. The main difference between to dynamically typed APIs
like Pandas is that you once have to tell the compiler about your schema -- but
then you can fully benefit from type-safety in the processing pipeline. Let's
say your CSV has columns _name_ (string), _age_ (int), _height_ (float),
_birthday_ (date), then your code would look like this:
const schema = [
col(StrCol, "name"),
col(IntCol, "age"),
col(FloatCol, "height"),
col(DateCol, "birthday") # DateCol not yet implemented, but coming soon
]
let df = DF.fromText("data.csv.gz").map(schemaParser(schema, ','))
What happens here, is that the `schemaParser` macro builds a parser proc which
takes a string as input and returns a named tuple of type `tuple[name: string,
age: int64, height: float, birthday: SomeDateTimeTypeTBD]` (note that this
allows to generate highly customized machine code, which is why the parser can
be much faster than generic parsers). So yes, the data frame only holds a
single type, but that type is heterogenous, and you can extract the individual
"columns" back by e.g. `df.map(x => x.name)` giving you a `DataFrame[string]`
instead of the full tuple.
Having to specify the schema might look tedious from a Pandas perspective. But
the big benefit is that you can never get the column names or types wrong. In
Pandas you see a lot of code which just says `def preprocess_data(df)`, and it
is neither clear what `df` really contains nor what assumptions
`preprocess_data` makes on the data. This can be solved by extensive
documentation & testing, but is still difficult to maintain in big projects.
With a type safe schema the assumptions about the data become explicit in the
code, and the compiler can ensure that they are satisfied.
Global aggregation is already available. You could do for instance `df.map(x =>
x.age).mean()` to get the average age. There is also `reduce`/`fold`, which
allows to implement custom aggregation functions. What's still missing is
`groupBy` and `join`, but they are high priority for me as well, so I hope I
can add them soon.
Re: Designing a data frame API
@krux: Thanks for the feedback! I was thinking about your `dataLanguage` idea as well. But I don't know if this would allow good composability in larger projects, i.e., you want data frames to be something that can be passed around etc. What I can probably do in the end is to write a macro which takes the iteration body: iterate(dataFrame): echo x and the macro would analyze the processing pipeline and generate a single for loop internally. The good thing is that the user side of the API only exposes transformations and actions. So it doesn't really matter for now if I use closure iterators, and I can still switch the iteration logic internally later. And the good news is that I get exceptionally good performance with the closure iterator approach already (I pushed a few first [benchmarks results](https://github.com/bluenote10/NimData#benchmarks)). I have optimized the CSV parsing macro a little bit, and CSV parsing is now a factor of 2 faster than Pandas, which is known to have a very fast parser. As expected for data which is still too small for Spark to shine, Nim is faster by a factor of 10 (although Spark already runs on 4 cores). I also made some good progress in terms of features and updated the documentation a lot, so this is reaching a state where it is actually pretty much usable. @perturbation2: Yes very good point, and thanks for the link. If Nim will not feature multi dispatch this might become a problem. For now I don't need multi dispatch for the higher order functions like `map` though (and maybe I never will), so I will simply stick to using a proc (which I hope will continue to work). I also understand issue 3 now, for the record: The compiler is basically complaining that the base method has a provable lock level of 0, while one of the overloaded methods calls a procvar, and potentially, this procvar can lock. The locking system needs to know lock levels at compile time though, which is ruined by the dynamic dispatching in this case. There are two ways out: Convince the compiler that the procvar does not lock, or tell the compiler in the base method that there will be overloads of unknown lock level. The former would be nicer, but I can't really get it to work for now -- for the latter I have opened a tiny PR to allow doing that in Nim.
Re: Function v. Writing operations
That doesn't really make sense. The for loop itself increments the variable. You can't increment it because it is immutable. And you need a colon and a lower bound.
Re: Documentation for structures created with macros
If I remember correctly the difference between generating docs with either `nim doc ` or `nim doc2 ` is that the latter includes stuff from macros. A quick check seems to confirm this: import macros macro test(): untyped = let testproc = quote do: proc someProc*(param1: int) = ## Documentation here echo "test" echo testproc.treerepr result = testproc test() Note that the macros module documentation mentions that the content of comments is not yet part of the AST (it appears as a `nnkCommentStmt` as the very first element of the body). But apparently that is not an issue, because it can be created from quote (or a template + getAst probably).
Designing a data frame API
I finally found some time to work on something I wanted to start immediately
when I discovered Nim: A data frame API allowing lazy out-of-core data
analytics. You can find a first working prototype over at GitHub (under the
uninspired name [NimData](https://github.com/bluenote10/NimData)). So far I'm
pretty happy with what is possible already now. In fact I got carried away a
little bit when creating the readme example based on this Bundesliga data set .
Quick motivation: The project is targeted between the worlds of Pandas/R and
map-reduce successors like Spark/Flink. In the Pandas/R world, the general
assumption is that your data fits into memory, and things get awkward when the
data size exceeds this threshold. Spark/Flink offer a different API design,
which makes distributed out-of-core processing easier, and many companies are
moving in this direction now. However, this can be a heavy investment because
Spark/Flink basically require that you build and operate a cluster (=>
costly/complicated) -- the performance of running in local mode is often not so
great (partly due to required abstractions + JVM). NimData uses a very similar
API design allowing out-of-core processing, but for the time being, the focus
is just on running locally. In other words, the target is "medium" sized data,
i.e., data which may not fit into memory entirely, but does not justify to
invest in a cluster. I haven't started benchmarking yet, but I could imagine
that Nim's native speed might even keep up with a small cluster for certain
problems.
In terms of the design I'm at a point where I could use some help from the Nim
veterans to verify if this is a good path. I went for a design based on dynamic
dispatching, and I was facing some of problems related to both method dispatch
and iterators. The biggest showstopper was that I could not mix generic and
specific methods (which didn't allow to actually read from files ), but then I
found the work-around documented in
[#5325](https://github.com/nim-lang/Nim/issues/5325). But there are still some
issues left, and I'm not sure if they are either a mistake on my part or a
known/unknown Nim issue. Some of the issues are tricky, because they disappear
in isolated examples. I have prepared the following minimal example which
illustrates both the general design and the outstanding issues:
import times
import typetraits
import strutils
import sequtils
import future
import macros
type
DataFrame[T] = ref object of RootObj
CachedDataFrame[T] = ref object of DataFrame[T]
data: seq[T]
MappedDataFrame[U, T] = ref object of DataFrame[T]
orig: DataFrame[U]
f: proc(x: U): T
FilteredDataFrame[T] = ref object of DataFrame[T]
orig: DataFrame[T]
f: proc(x: T): bool
proc newCachedDataFrame[T](data: seq[T]): DataFrame[T] =
result = CachedDataFrame[T](data: data)
#
-
# Transformations
#
-
# Issue 1: I'm getting a deprecation warning for this function:
# `Warning: generic method not attachable to object type is deprecated`
# I don't understand why it is not attachable, T and U are both unambiguous
# from the mapping proc. Is this a showstopper, because it will break
# in a future version of Nim?
method map[U, T](df: DataFrame[U], f: proc(x: U): T): DataFrame[T] {.base.}
=
result = MappedDataFrame[U, T](orig: df, f: f)
method filter[T](df: DataFrame[T], f: proc(x: T): bool): DataFrame[T]
{.base.} =
result = FilteredDataFrame[T](orig: df, f: f)
#
-
# Iterators
#
-
# Issue 2: I don't understand why this dummy wrapper is required below.
# In the `for x in ...` lines below I was trying two variants:
#
# - `for x in it:` This gives a compilation error:
# `Error: type mismatch: got (iterator (): int{.closure.})`
# which is surprising because that is exactly the type required, isn't it?
# - `for x in it():` This compiles, but it leads to bugs!
# When chaining e.g. two `map` calls the resulting iterator will
# just return zero elements, irrespective of what the original
# iterator is.
#
# For some strange reason converting the closure iterator to an inline
# iterator can serve as a work around.
iterator toIterBugfix[T](closureIt: iterator(): T): T {.inline.} =
for x in closureIt():
yield x
method iter[T](df: DataFrame[T]): (iterator(): T) {.base.} =
quit "base method called (DataFrame.iter)"
method iter[T](df: CachedDataFrame[T]): (iterator(): T) =
Re: Portaudio example doesn't work with threads:on
Maybe not the most helpful comment, but I can confirm that this works without problems on Linux. I only have to increase the buffer size to avoid underruns -- but has nothing to do with threads:on. In a more complex scenario, it's important to call `system.setupForeignGc()` at the top of the callback to get proper GC. But the saw_out example does not require GC in the callback, so won't help either. To track down the problem it would be interesting to build a minimal C example which sets up a p-thread like portaudio does, and calls a callback. If this minimal example fails as well, we would at least have something clear to work on (=> issue tracker).
Re: How to compile a shared library for JNI?
Yes, the traceback from within SBT indeed looks strange. To some degree this is a result of the forked VM. When running standalone or also with the pure Java example, there is no Java traceback at all, because the JVM segfaults straight away. I was just stumbling over [this comment](http://stackoverflow.com/a/28445066/1804173): > There is one catchy situation in JNI code: when such a code blocks SIGSEGV > signal e.g. because it blocks all signals (quite common approach in threaded > C code how to ensure that only main thread will process signals) AND it calls > 'back' Java VM (aka callback) then it can result in quite random > SIGSEGV-triggered aborting of the process. And there is virtually nothing > wrong - SIGSEGV is actually triggered by Java VM in order to detect certain > conditions in memory (it acts as memory barrier … etc) and it expects that > such a signal will be handled by Java VM. Unfortunately when SIGSEGV is > blocked, then 'standard' SIGSEGV reaction is triggered => VM process crashes. I have no idea how signal handling works in detail and will have to read up on it, but maybe something along the lines: The JVM trying to "communicate memory barriers via SIGSEGV" (wtf?) and the Nim signal handler catching it. Is there an easy way to test this? Like disabling Nim's signal handler altogether?
Re: How to compile a shared library for JNI?
I'm on Ubuntu 14.04, but with a more recent JVM (1.8.0_74-b02). Nim version is
the last tagged release 0.15.2.
Regarding the repeating: You could simply use that other `run.sh` script and
replace the java command by `sbt run`. Or if you want to avoid the load up
time: Add `addSbtPlugin("com.github.tkawachi" % "sbt-repeat" % "0.0.1")` to the
`project/plugins.sbt` and use `repeat 100 run` in the SBT shell.
Re: better example code for tutorial partI Iterators
You can: * Fork the Nim repository on Github: [https://github.com/nim-lang/Nim](https://github.com/nim-lang/Nim) * Edit the file doc/tut1.rst to your liking * Use "create pull request" to contribute your modification
Re: How to compile a shared library for JNI?
It varies a lot: When running the Scala example from the SBT shell it is maybe 1 out of 5. For the Java example on the debug branch I'm using the `run.sh` which simply performs 1000 iterations and terminates on a crash. I'm usually crashing within <200 iterations, sometimes in the first 10, max was maybe 400. When replacing the .so by the plain C library I'm reaching 1000 iterations without a crash. But I'm still not sure if it may be a JVM bug, which is only triggered by something in the Nim implementation because: * The traceback always points to `VM_Version::get_processor_features()` \-- with true stack corruption you would expect more random tracebacks, right? * When running a longer computation on Nim side, the crash always seems to happen after the function has returned.
Re: How to compile a shared library for JNI?
Good idea to emit `NimMain();`, but unfortunately it still crashes. I was wondering if it is due to either the function signature or maybe the `printf`. But using exactly your function does also crash (with a lower probability though, requires >> 100 iterations).
Re: How to compile a shared library for JNI?
@Krux02: Everything prepared here: [https://github.com/bluenote10/JniScalaNimExample](https://github.com/bluenote10/JniScalaNimExample) The master branch contains a SBT + Scala setup, including a customized build task (coupled to "compile") and source monitoring so that ~test / ~compile / ~run are fully functional. The debug branch is the minimal example (in Java for better reproducibility) which does segfault for me. But for instance @yglukhov already confirmed that he is not getting segfaults on MacOS with this example. @Araq: That was also a suggestion by @yglukhov, but unfortunately it also crashes. Any pointers in which direction you would continue testing?
Re: How to compile a shared library for JNI?
Actually I was doing that, I only made a mistake simplifying the problem for demonstration purposes. On the master branch I have set up the whole SBT tool chain using the sbt-jni plugin, which comes with a javah interface for Scala. In my original version I obtained the correct signature from that. By the way, I'm actually pretty happy how well Nim integrates with the Scala+SBT world. I just verified: Using the correct for the simplified example does not make a difference.
