Re: Nim vs. Python & Groovy (string splitting): Why is string splitting so slow in Nim?
I want to encourage on your journey with Nim. Nim can be a challenge especially if you are coming from a dynamic language like Python. Because Nim has a nice and reasonably friendly syntax for most cases. It is not a Python-like language. You can not approach Nim like Python. It is a system's level language. And as such it can offer you things Python can not. But it will also require from you things Python does not. In my journey with Nim, I have approached it from a high level. I am still learning my rhythm for developing with Nim and my style. The challenging part for a person who is primarily coming from a higher level language is that Nim on the surface feels pretty high level. But then there are times when it pulls you down lower than you are accustomed to. You may start with more naive approaches to problems, similar to how you might of in Python. It may work. It may be sub-optimal. But there are solutions. This forum is very helpful. People here will help direct you toward a more idiomatic Nim solution. Once again, it may require more from you than Python. But it will give more back. Eventually you will become comfortable in Nim and these issues will become fewer and fewer. Nim is far from slow. In my app I am parsing GBs of data. I started with a split-like solution. Moved to Nim's csv. Then I changed the format of my data to fixed width columns. Now I read in x number of bytes at a time and process. I can now process 2GB of data with over 77million rows of data in 24 seconds. That is if I compile with -d:release. If I do not then the data parsing is over 3 minutes. You will want to use debug mode while you are developing so that you have debug information when you encounter problems. But if you are ready to release or simply test performance. You will want to compile with -d:release. There are also time when you can pass C compiler flags into Nim via -passC: and possibly get even more amazing performance gains. It greatly depends on your app. Be encouraged. Ask questions. When Nim pulls you under to a lower level than you understand. Ask. Get help. Keep going. It is worth it.
Re: Dead Code elimination
I haven't yet written the part which parses the input config file. I wasn't sure if I would need to do anything special. But as I think on it, you are right. It is definitely a case of premature concern. Thanks everybody who replied.
Re: Dead Code elimination
Yes. I had read that post. But I didn't find anything conclusive in the thread. It is also 4 years old I didn't know what has or has not changed. I currently do not find any documentation for a deadCodeElim pragma. I could have overlooked it. But I do not see it in the manual or the compiler or internal guides. Thanks.
Dead Code elimination
Hello, I have searched the Nim site and this forum trying to read up on deadCodeElim. I am not finding sufficient information for me to understand. Pointers to docs are always welcome. Looking through the manual and compiler or internal guides, I did not find much or overlooked something. I did a quick scan and search for dead. I am writing an application that will have objects that I write in my modules that will not have any objects created at compile time. I do not want any of these eliminated. They will be created at runtime when a configuration file is read into the app. I am happy for deadCodeElim on any of my standard library imports and such. Is their a way to protect specific modules from deadCodeElim and allow others? What is the proper way to accomplish this? Thanks for any help, knowledge and wisdom.
Re: Trying handle HSlice
Thanks for the reply. I will firmly admit that learning generics is tough. The documentation is slim. And I don't know where a nice set of examples exist. That is why I am pressing through displaying my ignorance and hoping for some help. My apologies for the very simple sample code. I was trying to provide enough code to convey the problem. But not so much as to bog down in unnecessary information. I will endeavor to provide better examples and sufficient information in the future. You are correct I did not provide a valid constructor I did not feel it necessary for the question. And yes I do understand that a seq is not fixed sized. But my object is. It will never increase the size of the seq which it contains. The fact that it is a seq is an implementation detail. I would use an array. But I do not have sufficient information at compile time to create an array. I don't know that there will ever be any users of the slice operation of my object. But I wanted to learn how to do it and code it. I believed it would be another step to understanding generics. And I was correct. You did provide me with enough information to make it work. I am writing a trading app. I think Nim is a great language for such an app. I did not start writing the app in Nim but in Smalltalk. Performance was not the best but sufficient. Stability is what became the problem. When I had an array of 5-100million floats and the memory was 1gb+. I was going to finish the app and get version 1 running. But with the stability problem. I decided to go to what I was going to write version 2 in. It will take me a little longer as I am far and away not as familiar with static typing, static compilation and Nim in general. In trading you have a continual stream of time series data. Which is a set of values containing (datetime, open, high, low, close) for the specified time period. Then you operate on those values. A common item is a fixed sized slice of the data from the most recent, back a specified number of periods. Commonly this is done in a multi-dimensional array maintaining a large number of periods of data. And on the column you need a slice of `mydata[latestIndex - numberOfPeriods .. latestIndex]`. I have chosen not to go that route but simply maintain only the data which which is necessary for my specific strategy. For your complete understanding of what I am actually doing. Not a sample. I will provide the complete code below. Should anybody like any of the code. I hereby release it under the MIT license. The MovingArray below is somewhat like a FIFOQueue that is fixed in size. There may be something out there like this or better than this. I am not an expert in programming or algorithms. So my implementation may be naive or better implementations exist. But I will display my ignorance. I have a lot to learn. Thanks again. ##MovingArray ##(c) copyright 2019 Jimmie Houchin ##MIT license type MovingArray*[T: int|float] = ref object ##MovingArray is an array type object with a fixed number of values. ##It maintains a small set of state variables which are update each push() values*: seq[T] size*: int index: int sum*: T high*: T low*: T proc newMovingArray*[T: int|float](size: int, tp: typedesc[T]): MovingArray[T] = result = MovingArray[T]( size: size, values: newSeq[T](size), sum: 0, index: size-1, high: low(T), #guarantees first value will replace initial value low: high(T) #guarantees first value will replace initial value ) proc `[]`*(ma: MovingArray, index: int): int|float = var i = ma.index + index if i > ma.size-1: i = i - ma.size result = ma.values[i] proc `[]`*[T: int|float](ma: MovingArray[T], hs: HSlice): seq[T] = result = newSeq[float](hs.b - hs.a + 1) var index = 0 for i in hs: result[index] = ma[i] index += 1 proc push*[T: int | float](ma: MovingArray, n: T): T = if ma.index == ma.size-1: ma.index = 0 else: ma.index += 1 result = ma.values[ma.index] ma.values[ma.index] = n ma.sum = ma.sum - result + n if ma.high < n: ma.high = n elif ma.high == result: ma.high = ma.values.max() if ma.low > n: ma.low = n elif ma.low == result: ma.low = ma.values.min() proc avg*(ma: MovingArray): float = result = ma.sum / ma.size Run
Trying handle HSlice
I am attempting to write a proc for `[]` which handles `HSlice`. I am failing to see how to convey to the newSeq the desired T type MyArray*[T: int|float] = ref object #Fixed size array type object with state. values*: seq[T] size*: int #... proc `[]`*[T: int|float](ma: MyArray, hs: HSlice): seq[T] = result = newSeq[T](hs.b - hs.a + 1) # ... # I want to be able to do ... var ma = newMyArray(...) let sliceOfMA = ma[5..10] Run Any help, wisdom or understanding greatly appreciated.
Re: Problem catching SslError
Ah okay, silly mistake on my part. That fixes it. I have another app which uses the code with the `SslError` and it compiled without problem but I was using the `-d:ssl` flag to compile it. I did not notice I was doing it differently here. Thanks for your patience with my mistake.
Problem catching SslError
I am trying to catch SslError in some HttpClient code.
import net, httpclient
try:
var client = newHttpClient(timeout=500)
except SslError:
echo("caught SslError")
#Gives this error
#~/ntplayground2.nim(5, 8) Error: undeclared identifier: 'SslError'
Run
To my understanding SslError is in the net module.
[https://nim-lang.org/docs/net.html#SslError](https://nim-lang.org/docs/net.html#SslError)
I do not understand why it is undeclared? Or how to resolve the problem.
Thanks.
Re: What text editor are you using for Nim?
Thanks for the reply. I didn't know that their was a nim.kak. Interesting story. I don't know how I would classify myself. I am a long time Smalltalker. I have struggled entering the world of editors and have kind of simply settled for a reasonable IDE/Editor rather than committing the time to learning Vim or Emacs. So currently that is Atom. I am not a fan of bloat of modern apps. I am also doing more server side apps. I would like a ssh, terminal friendly editor. Vim is definitely one. It seemed that Kakoune would be somewhat an evolution of that. If I were to learn Vim first, am I going to have to unlearn or relearn much should I at a later date move to Kakoune? Thanks for the advice.
Re: Trouble creating type containing generic array
Thanks for the advice. I will look into it.
Re: Trouble creating type containing generic array
My apologies for not being more clear. I could easily simply create and pass around the plain seqs. But that defeats the purpose. The purpose is to maintain state and knowledge of the array or seq that a smarter object can do. For example in time series data that I collect as time goes on or on historical data that I operate on as if I am in the moment. It is far less expensive to maintain state of a seq than to perform operations on it. Especially if those operations are being done over and over. I tried for a minimal example that would teach me what I needed to know to fix my understanding and solve my problem. However it is not a complete example. An easy example is if I had a seq of which I wanted to know the sum, average, max and min. All these things are easily achieved in a seq or array and computed when desired. They can also become expensive. What is the cost of sum, max and min over a arbitrarily large array? Verses the cost of maintaining a sum, max and min. And then you multiply that cost by every object which has a reference to that seq. That is what I am trying to achieve. Nim is amazing. Its performance in those computation is outstanding. However, if I can avoid them in the first place. I can do better. Creating such an object even in a dynamic language makes knowing the sum, max and min instant even over a multi-million valued array. Then the question might be why have the array not simply maintain state. The array is part of the state. And I do not know that there are no other requirements which can only be met by the array itself. Hope this helps. Thanks for engaging in the conversation.
Re: Trouble creating type containing generic array
Thanks. That works and got me going. I wasn't trying to pass types as a string so much as knowledge about the type. I am still working on understanding generics. I have not found much documentation regarding generics. I have the book and I have read the Tutorial II. Most documentation is regarding defining a proc not a type. Is there documentation I am overlooking? Or is there a good source of source code which would have good examples that are comprehensible by mortals. :) Again thanks.
Trouble creating type containing generic array
I am trying to create a type which is basically an Array with additional information. I want to use only [int | float] for the Array. type MyArray* = ref object length*: int index: int arr: seq[int|float] proc createMyArray(length: int, tp: string): MyArray = if tp == "int": let ttype = int elif tp == "float": let ttype = float result = MyArray( length: length, index: 0, arr: newSeq(ttype)[length] ) var ma = createMyArray(int(rows), "float") Run The code is probably naive as I am still learning. The MyArray type compiles if everything below is commented out. As soon as I uncomment, it gives me this error. /home/jimmie/Dev/Nim/ntplayground.nim(125, 19) Error: invalid type: 'int or float' in this context: 'proc (length: int, tp: string): MyArray' for proc Run Any help, wisdom or understanding greatly appreciated.
Re: What text editor are you using for Nim?
I have been briefly exploring Kakoune. I like the idea of a less bloated editor. Currently I have been using Atom. I have not yet really gotten into Vim, or Emacs. I wish it were available for Android. I don't see Nim support. Are you using the lsp plugin? Thanks.
Re: Dynamic Tuple creation
Thanks. I will take a look.
Re: Dynamic Tuple creation
I was not sure if there were a way or not. Tuple is what I want for production code. By then I will know all the details. During exploration and experimenting JSON will work fine. Thanks for the information and suggestion. The link was educational.
Re: Struggling with writing an iterator
I have read that before. I don't quite understand all of it in relation to how
to write an iterator and call it.
However, I went and put back the {.closure.} in my iterator and it works if I
call like this:
var stream = pricestream
for p in stream("oanda", EUR_USD):
...
Run
What I don't understand is something like this:
iterator walkDir*(dir: string; relative=false): tuple[kind: PathComponent,
path: string] {.
tags: [ReadDirEffect].} =
...
when nimvm:
for k, v in items(staticWalkDir(dir, relative)):
yield (k, v)
...
#And walkDir is called just like just like a proc and it contains no
{.closure.}
for file in walkDir(datapath / source / $instrument / "Prices"):
paths.add(file.path)
...
Run
Thanks for at least getting me working even if lack understanding.
I don't understand why the intermediate step of assigning the iterator to a
variable and then providing the variable with the parameters.
Thanks again.
Struggling with writing an iterator
I am writing an iterator which calls walkDir and iterates over all of the files
in the dir creating objects from those files. I have a proc which reads the
file and creates a seq of the objects. It works fine. Amazingly so, on my
machine Nim will read one of these files and create almost 400k objects in 40ms.
I want an iterator which iterates over each of these files, serving each of its
objects one at a time.
iterator pricestream*(source: string, instrument: Instrument): Price =
## Iterator for all historic prices for the source, and instrument.
var paths: seq[string]
for file in walkDir(datapath / source / $instrument / "Prices"):
paths.add(file.path)
paths.sort(system.cmp[string])
for path in paths:
for price in pricesFromCsv(path):
yield price
# If I call like this:
var stream = pricestream("oanda", EUR_USD)
for p in stream:
...
# I get this:
# ntplayground.nim(46, 27) Error: attempting to call undeclared routine:
'pricestream'
# If I call like this:
var stream = pricestream
for p in stream("oanda", EUR_USD):
...
# I get this:
# ntplayground.nim(47, 18) Error: type mismatch: got
# but expected one of:
# iterator items[T](a: set[T]): T
# ...
Run
Any help in understanding how to write and call an iterator (this iterator)
would be greatly appreciated.
Re: Understanding performance compared to Numpy
Thanks for all the replies. I will have to take some time later to analyze the more idiomatic code above and learn. I understand my code may not appear to make sense. It's primary purpose is simply to provide somewhat of a stress test of what I might commonly do in various methods/functions. I like the idea of a compiled solution which is smaller and light weight verses Numpy and such. My requirements are reasonable performance and do not require the breadth or depth of the Numpy stack. My code was simply copied and pasted from a playground.nim file where I explore and play around trying to learn and understand. Simply compiling my code with the above nim c -r -d:release --passC:-march=native --passC:-ffast-math ntplayground.nim worked wonders. My code as existing now runs in under 30 seconds. Performing better than the Python/Numpy code. Yay! I am very happy that Nim properly understood performed as I had hoped. Hopefully I will have time to learn from this thread on Sunday. Time to spend time with the family. Thanks again.
Understanding performance compared to Numpy
Hello,
I have some code that I am trying to understand why it is slower than
Python/Numpy. I am not a C programmer and Nim is my first real attempt to learn
a systems level, statically compiled language. But to my understanding Nim
should approach C speeds to a certain extent. If what I present can be improved
please let me know. I want to learn and I want to use Nim correctly.
I know Python/Numpy has been well optimized for certain tasks. I understand my
Nim code may be naive. I would love to see the Nim code that competes with the
Numpy code. I want to understand how to write it well.
Ubuntu 18.04 x86-64
Python/Numpy perfoms this at 0:50.0. Nim performs this at: 1:43.151
When I remove the sum() code Nim is 5x faster.
I need fast summing and averaging of slices of arrays. Numpy does this
exceptionally.
I know about Arraymancer but am lost at looking at it. I do not do high level
maths. I only need simple arrays and accessing, summing, averaging, slicing and
doing similar on the slices. I do not need any matrix type operations.
I am okay with doing what I need with Arraymancer should it be the solution to
speed up my app. But I am currently clueless on how to do so with the below
code.
Any help in accomplishing these goals and competing reasonably with
Python/Numpy would be greatly appreciated. I can easily complete my app in
Python. And I may do so. But I would like either initially or app v2.x to be
able to do it in something like Nim.
I understand very well I may have done something stupid and/or naive. I am
willing to learn.
Thanks.
#latest nim dev
proc pricesFromGainCsv*(fullpath: string): array[337588, float] =
let csvfile = open(fullpath, fmRead)
let contents = csvfile.readAll()
let lines = contents.splitLines()
for i in 1.. 0:
let parts = line.split(",")
result[i-1] = parseFloat(parts[5])
csvfile.close()
var ctime = getTime()
echo("time: ", ctime)
var cpustart = cpuTime()
#http://ratedata.gaincapital.com/2018/12%20December/EUR_USD_Week1.zip
let gaincsvpath = "/home/jimmie/data/EUR_USD_Week1.txt"
let prices = pricesFromGainCsv(gaincsvpath)
var psum = 0.0
var pcount = -1
var pips = 0.0
var psumsum = 0.0
var farray: array[prices.len-1, float]
for price in prices:
pcount += 1
farray[pcount] = ((price * price) * ((pcount+1) / prices.len))
pips += price
psum = farray.sum()
psumsum += psum
echo("pcount: ", $pcount, " pips: ", $pips, " farray[^1]: ", farray[^1],
" psum: ", $psum, " psumsum: ", $psumsum)
echo("clock: " & $(getTime()-ctime) & " cpuTime: " &
$(cpuTime()-cpustart))
#pcount: 337587 pips: 383662.562762 farray[^1]: 1.295654754912296
psum: 218237.9662717213 psumsum: 24517634293.9183
#clock: 1 minute, 43 seconds, 151 milliseconds, 497 microseconds, and 564
nanoseconds cpuTime: 102.814928774
Run
#python 3.7 latest numpy
#http://ratedata.gaincapital.com/2018/12%20December/EUR_USD_Week1.zip
stime = time()
fpcsv = "/home/jimmie/data/EUR_USD_Week1.txt"
count = -1
pips = 0
psum = 0
psumsum = 0
f = open(fpcsv,"r")
csvlines = f.readlines()[1:]
f.close()
csvreader = csv.reader(csvlines)
pasize = len(csvlines)
parray = np.ndarray(shape=(pasize,),dtype="f8")
for tid,d,pair,dt,b,a in csvreader:
count += 1
price = float(a)
parray[count] = ((price * price) * ((count+1) / pasize))
pips += price
psum = parray.sum()
psumsum += psum
print(count, pips, psum, psumsum, time()-stime0)
#337587 383662.562762 218239.26158885768 73674955841.14886
50.0512046814
Run
SslError: ssl3_write_pending:bad write retry
Hello, In an app I am writing I am getting the following error. This is occurring inside code which when encountering an exception will sleep(100) and retry. After 10 retries (1 second) it will close the client and create a newHttpClient. Rather than catching this exception the app just stops, goes silent for 10+ minutes and then quits with the below. I am not sure how to handle this. Any help greatly appreciated. Current Exception: SslError Current Exception Msg: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry Current Exception: SslError Current Exception Msg: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry Current Exception: SslError Current Exception Msg: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry ... Traceback (most recent call last) mycode... httpclient.nim(1235) get httpclient.nim(1227) request httpclient.nim(1204) request httpclient.nim(1183) requestAux net.nim(1379)send net.nim(1367)send SIGPIPE: Pipe closed. Run
Re: HttpClient networking problem
Mark this one as Solved. I want to thank you all for pushing me to persist and for providing me with some information to help solve the problems. What helped was showing me to use the timeout. Once that happened the errors happened in a timely manner. And then I had to discover a few places where I forgot the timeouts. Now it fails and I catch it and keep going. The one thing I don't know is what is the best way to determine that the current connection has closed and I need to create a newHttpClient. I see in the docs that the HttpClient.connected var is private. If there is a good way to determined the connection closed status, I would appreciate it. Again thanks.
Re: HttpClient networking problem
Thanks for the reply. I don't know if that code makes any difference or not. But it is correct and I made the change. And whether or not that change did anything when I ran the code again I did get an error right up front. And then I noticed an error in the code I posted. I have the "while" loop inside the try/except. It never had a chance to loop. I moved it outside the loop and the code now does what I want. It loops indefinitely. However, I went back and looked at my original code. And it did not have such a problem. However, I did notice a different logical error on my part. But the api is only open 22:00utc Sunday to 22:00utc Friday. I will test tomorrow on Sunday. Thanks.
Re: HttpClient networking problem
Hello,
My apologies that terminal session is gone.
Here is what I am attempting. Here is some code that I would like to see run
indefinitely.
I would think that while I have it running and I turn off my network that at
some point I would see an Error raised. I almost never do.
My naive understanding of what I wrote is that if any Errors would work up to
my code then it could loop correctly and checking the connection and eventually
when the network is restored continue getting Google.
I am not understanding why this isn't working and why I am not seeing the
Errors.
Maybe this is enough information that you who are far, far more knowledgeable
than I can see the error in my ways and put me on the right path. Wisdom and
knowledge gratefully accepted.
Thanks.
import httpclient, httpcore, net, times, strutils
from os import sleep
let
resetDuration = initDuration(seconds=2)
deciSecondDuration* = initDuration(milliseconds = 100)
qtrsecondDuration* = initDuration(milliseconds = 250)
var
client = newHttpClient()
lastConnection = getTime().utc
proc resetHttpClient() =
if (getTime().utc - lastConnection) > resetDuration:
# Probably a new timeout. We have not yet experienced a long outage.
# We may however be entering an extended outage.
# Creating the new clients seems to use up lots of CPU.
# I want to do that as little as possible.
try:
client.close()
except:
echo("Attempted to close clients. Probably do not exist.")
echo("Current exception: ", getCurrentExceptionMsg())
client = newHttpClient(timeout=500)
proc getGoogle() =
resetHttpClient()
var dt = getTime().utc
let enddate = dt + initDuration(days = 1)
try:
while dt <= enddate:
echo(dt)
echo(client.get("http://www.google.com;).body[0..14], " ",
getTime().utc)
let diff = getTime().utc - dt
if diff < deciSecondDuration:
sleep(diff.milliseconds.int)
dt = getTime().utc
except TimeoutError, IOError, OSError:
# I naively think I would see this thrown or the plain except below.
# But I almost never see an Error raised.
echo("Current Exception: ", getCurrentException().name)
echo("Current Exception Msg: ", getCurrentExceptionMsg())
echo("Sleeping for 1 seconds at: ", getTime().utc)
sleep(1000)
resetHttpClient()
except:
echo("Current Exception: ", getCurrentException().name)
echo("Current Exception Msg: ", getCurrentExceptionMsg())
echo("Sleeping for 1 seconds at: ", getTime().utc)
when isMainModule:
echo("Executing network_test")
getGoogle()
Run
If I turn of my network and then Ctrl-C after a few seconds I get this:
\--(Is there a style code for something like this terminal session output?)
2018-11-30T22:35:27Z 2018-11-30T22:35:27Z ^CTraceback (most
recent call last) proxyexe.nim(62) proxyexe proxyexe.nim(49) main
osproc.nim(1136) waitForExit SIGINT: Interrupted by Ctrl-C. Traceback (most
recent call last) network_test.nim(48) network_test network_test.nim(33)
getGoogle httpclient.nim(1235) get httpclient.nim(1227) request
httpclient.nim(1204) request httpclient.nim(1189) requestAux
httpclient.nim(1024) parseResponse net.nim(1312) recvLine net.nim(1272)
readLine net.nim(1069) recv net.nim(1056) readIntoBuf net.nim(1052) uniRecv
SIGINT: Interrupted by Ctrl-C. Error: execution of an external program failed:
'/home/jimmie/Dev/Nim/network_test/network_test '
After almost 16 minutes I get this output.
2018-11-30T22:44:21Z 2018-11-30T22:44:21Z 2018-11-30T22:44:21Z
Current Exception: OSError Current Exception Msg: Invalid argument Sleeping for
1 seconds at: 2018-11-30T23:00:06Z
Re: HttpClient networking problem
You are probably right. I probably do need to drop a level. Since yours and @dom96 replies I have experimented some more. I have tried catching, TimeoutError, IOError, OSError. I have reduced the timeout in the HttpClient to 500millis. Sometimes it catches the errors. Sometimes none show up in my try/except. Most of my experience the higher level library isn't totally protected from the errors and gets a chance to handle them. Which is what I was thinking I was going to get here. SIGINT: Interrupted by Ctrl-C. SIGPIPE: Pipe closed. Some sort of IO Error I have a few parallel implementations of this app in various languages to see which I like the best. I want to thank everyone for their help. I think at this present time and the schedule I have to implement this app, that Nim is not the best match for me. This is not anything against Nim. Simply that Nim and I are not an ideal team at the moment. Thanks.
Re: HttpClient networking problem
I have a plain except:. It should catch everything. But nothing is being caught. It isn't a server problem. I am testing what happens to the program if there are network problems like when a network goes down briefly. In this instance. I am simply turning off my wifi while it is running. I was hoping to find what Errors get raised that I could catch and simply have a loop which sleeps a little while, then tries again, until I have success. I can work on writing a sample app. It shouldn't be difficult. But it can't be the one I am using at the moment because it is for a secured api which requires an account and credentials. But again the key part is nothing is happening when I disconnect the network. It just goes comatose. No errors. The server side is fine. When my network is up, the app works flawlessly. If the outage is short enough, 2 minutes and under from what I have briefly experimented with. It simply resumes. But my app has not been informed of any issues and I have missed ??? amount of data during the outage. I am at a loss on how to make this fault tolerant and how to debug. Thanks.
HttpClient networking problem
Hello,
I am writing an app which requests data from a REST API multiple times a
second. At the moment I am testing in a loop which collects data, sleeps until
the next .1 of second and then continues collecting data.
Below is some slightly modified, abbreviated and cleaned up code which does
what I want.
It works fine most of the time. I am wanting to make the code robust against
network failures.
I am not sure how to do that. When running it on my laptop I will disconnect my
wifi. It produces no output while the wifi is disconnected. I get no output
from the request which as to be expected. But I also get no errors. It just
sits idle. I don't like that. I can imagine there is some code built into
HttpClient that also attempts to be robust. However, I do not know how or when
it will fail.
If there is network failure I want to be able to resume as soon as it is back
up.
The app sometime seems to just go comatose. Even after the network is back up,
it does not resume, nor give any kind of exceptions. Which is why I wrapped
this proc in a try/except and put in many echo statments. How do I debug this
when disconnected? How do I make this robust to failure where I can resume
immediately after any sort of network failure when the network returns?
Any help and wisdom greatly appreciated.
Thanks.
var client = newHttpClient()
proc secondLoop() =
try:
dt = getTime().utc
let pathQuery = "my path and query string"
echo("before secondLoop1: ", getTime().utc)
let result = client.get(pathQuery)
echo("in secondLoop2: ", getTime().utc)
echo("print some of the result")
let diff = getTime().utc - dt
if diff < deciSecondDuration:
sleep(diff.milliseconds.int)
echo("after secondLoop3: ", getTime().utc)
except:
echo("exception message: ", getCurrentExceptionMsg())
Run
Re: Problem sleeping
Slaps self on forehead. Okay, I know see what you and the manual are saying. It seems obvious when you see and invisible if you don't. For whatever Type, such as int, simply using the Type as the function to do the conversion. Ugh! Such that, originalType.convertedType should work for anything where the conversion is implemented? Thanks for helping the old guy see the obvious.
Re: Problem sleeping
Thanks for the reply. It has been awhile since I read the tutorial. However, when searching [https://nim-lang.org/docs/lib.html](https://nim-lang.org/docs/lib.html) I do not find the int() procedure. I do not find any procedure answers the question of how to provide the right type. Yes I understood the error message. It was readily apparent. It was getting from where I am to where I want to be that I could not find a path. My question is and has been is there something I am missing from the standard documentation or is the standard documentation at present not at a place to provide answers like this. I still do not see where this is defined. And this is for a question we know the answer to. Let alone a question that I do not know the answer and have not asked. Nim has different challenges with regard to discoverability. In your OO languages you generally find all methods associated with a Type in the class definition. As they generally lack the ability to modify the class and add methods anywhere else. Smalltalk is a little different in this regard. Regarding Smalltalk and Python. I am a long time user of both. Python longer, but Smalltalk more. I enjoy Pharo. I am able to do my app in Pharo and may do so. I have this app partially in Nim, Python and Pharo. Most recent exploration in Python showed significant performance problems. So I wanted to explore Nim some more. Nim is very different from Pharo or Python. There is much I like about Nim. The things I don't like are not necessarily problems with Nim, but rather my lack of experience with Statically typed and statically compiled languages. That is something that I do want to learn over time. I love the idea of small, fast, memory efficient apps. I hate the current trend towards lazy, bloated and slow apps. I have explored many, many languages over the course of many years. I am not a professional programmer. I have a day job and a family. So like many people, time is limited. Nim looks like it hits a pretty sweet spot with its syntax, performance, etc. It's current or past state of maturity has been its most limiting factor for me. I have limitations in skill, and time to acquire said skill. However, Nim is getting close to the place where my skills and its maturity are able to start doing things. I am one of the people you refer to. Looking at my profile, I have been registered for a little over 3 years with 24 posts. Most of my reasons of stopping my attempt with Nim has been the combination of where I was skillwise and where Nim was maturity wise. But I have come and gone a number of times. I keep checking back. I think there are probably a lot of people like me. And as Nim matures and its ecosystem grows. The ability for people like me to use Nim will improve. I have also explored Rust, Kotlin, OCaml, C++, C. I really like Nim. I think Andreas has created a beautiful language. I think he made a lot of great decisions. And now that there is a little money behind Nim. I think the future is bright. Nim is different from what I am used to. But different is not bad, it is simply different. I take full responsibility for the problems being with me and not Nim. I am not assigning any blame Thanks for the suggestion of the IRC. I had forgotten about that. I am used to mailing lists. I will check out the IRC channel and try there first unless it is something more permanent which deserves post on the forum. Again, thanks for engaging in conversation. This is a good community. :)
Re: Problem sleeping
Thanks. I do need to read the manual. However, nothing I read there leads me to understand that there is a .int procedure that can be applied to an int64. Everything I have seen about converting to an int is a floating point number. I come from Smalltalk and Python and am used to an environment I can explore. In Smalltalk everything is available all the time. Discovery is relatively easy. How do I find the standard procedures available to operate on any type? In this case int64? Where do I find proc toInt(i: int64): int = ... convert int64 to int ... # or proc int(i: int64): int = ... Run I would love to be able to discovery the answer to the question I posted on my own and reduce the frequency in which myself or someone else asks a question whose answer is discoverable. I hope that makes sense. Thanks again for the help and replies.
Re: Problem sleeping
Thanks for all the replies. I thought it was a funny title also. I didn't have a better one. I looked for how to convert int64 to an int, but I did not find one. How would I discover this proc? I don't currently use a magical IDE like VSCode. I have not found it within me to use it. Thanks.
Problem sleeping
Hello,
I am trying to create a loop which do some stuff and when finished sleeps until
the next iteration at the top of the next minute.
I have the below code but I get a type error when trying to sleep.
import times, os
let minute = initDuration(seconds = 60)
let dt = getTime().utc
var dt2 = dt + minute
dt2 = initDateTime(dt2.monthday, dt2.month, dt2.year, dt2.hour, dt2.minute,
0, utc())
let dur = dt2-dt
let dt4 = getTime().utc
let sleeptime = dur.seconds*1000 + dur.milliseconds
echo("sleeptime: ", sleeptime)
sleep(sleeptime)
let dt5 = getTime().utc
echo("dt4: ",dt4)
echo("dt5: ", dt5)
Produces:
playground3.nim(17, 6) Error: type mismatch: got
but expected one of:
proc sleep(milsecs: int)
expression: sleep(sleeptime)
> exit status: 256
Run
If I comment out the call to sleep(), and run. I can take the resulting
sleeptime and manually place it in the code such as
sleep(16321)
Run
And it works just fine.
I do not understand why it doesn't work.
Any help and wisdom greatly appreciated.
Thanks.
Re: Error converting json to types
The below fails with the same error.
type
DateTime* = string
response = """
{
"datetime":"2016-06-22T18:41:36.042678456Z"
}
"""
var data = parseJson(response)
echo(to(data, DateTime))
Run
If I am doing something wrong. I am okay with that. But I don't see what it is.
I have this app mostly written in Python. I might simply need to finish version
1 with Python and revisit Nim when I have more time to be better at Nim.
I am in complete control of my app and all client side code. The only thing I
don't control over is the REST api coming from the third party servers. I am a
consumer and also producer of json to them in the requests.
Thanks for your help.
Error converting json to types
Hello, I have been working on an app which calls a REST api which returns json objects. I have created the types and when I attempt to convert I get this error. lib/system.nim(2833, 7) Error: unhandled exception: false Unable to process nnkSym DateTime It is probably a simple newbie error. But I am failing to see why it fails or how to fix. Any help greatly appreciated. Here is from what I see the pertinent code. type DateTime* = string Price* = object time*: DateTime ... proc test_prices(data: string): seq[Price] = let jsonNode = parseJson(data) echo($jsonNode) result = to(jsonNode, Price) Run Thanks.
Re: Introduction
Hello Göran, I have liked what I read about Spry. I look forward to giving it a try. I plan on doing so once I get up to speed and am comfortable with Nim. Hopefully I can join you on that journey. The family is doing well. I hope yours is as well. Jimmie
Re: Introduction
Hello, Many forums have a long running thread for introductions. So I am joining this thread. Hopefully others will continue in the future. I just wanted to say Hi and introduce my self. I am just learning Nim, but I have decided to make a commitment. I have read the tutorials and bought the book and am working my way through it. I have also joined the Bountysource where I could afford. I am not a professional developer, but would like to attempt some serious projects. One of the things that attracts me to Nim is the natural ability to interface with both C and C++. I have used lots of things which do okay with C, but C++ always required a C wrapper around the C++ which made it either difficult to near impossible. I have projects where I will need that ability. I have used Python casually for 20+ years. I have also been a long time user of Squeak and Pharo. I have known Göran Krampe for a long time before he was Göran Krampe. As a long time casual user of Python for a long time, I find the syntax of Nim to be comfortable. I think it is in some ways nicer than Python, but I do have to get used to the static typing. As a long time user of Squeak and Pharo, like Göran, I do miss the live environment. I would dearly love to see some things that direction in the future. Pharo has been my language and environment of choice the last several years. I am enjoying learning Nim. I like the features it provides. I like the efficiency and I do believe it makes for a nice balance between efficiency and elegance. I hope to be able to contribute back to the community when I am up to speed on using Nim. Thanks for this language and the reasonably unique set of features it has. Jimmie
