Here is an example to play with: https://gist.github.com/josiahcarlson/3250376
I have it running at curl http://csilo.com:8777 As I suspected, the default J gethttp won't work as it blocks the thread. You'd need to spawn this in another process and do the file reader load 'web/gethttp' gethttp 'http://csilo.com:8777/' I'll run the python example in a shell until it dies if you want to play with it. It's easy to get going on your own though python chunked_server_test.py On Wed, Jul 16, 2014 at 1:18 PM, Raul Miller <[email protected]> wrote: > I went to try and throw together an implementation, but the urls you gave > are protected, which means I couldn't debug the resulting code. > > You might want to stick with commercial offerings, or hire somebody to get > it working for you. > > Thanks, > > -- > Raul > > > > On Wed, Jul 16, 2014 at 10:48 AM, Francisco Gonzalez Velasco < > [email protected]> wrote: > > > Thanks for your help, > > > > I think this is far beyond my current abilities with J. > > > > Cheers > > > > > > 2014-07-15 13:34 GMT+02:00 Joe Bogner <[email protected]>: > > > > > I researched this a bit and here is what I found: > > > > > > 1. oanda appears to use chunked encoding for streamed requests[1]. Curl > > > will automatically handle this > > > 2. There are several examples[2] available in other languages to learn > > from > > > 3. The examples will trigger a new record based on the presence of the > > word > > > instrument in the stream as a delimiter > > > 4. J's web/gethttp will use curl or wget and will handle chunked > encoding > > > as a result, but I suspect it will wait for the connection to close > > before > > > returning any of the message > > > > > > The key decision to make in my opinion is whether to stream to a file > or > > to > > > parse everything in memory > > > > > > Streaming to a file likely makes sense in this scenario. There can be > > > streaming use cases where it's impractical to stream to a file because > > the > > > stream would consume all disk space -- then you'd have to come up with > > some > > > rolling file mechanism, which would not easily be handled by curl/wget > as > > > far as I know. > > > > > > Parsing in memory would require writing your own HTTP get utility. You > > > could take a look at Henry Rich's webio[3] which implements much of > what > > > would be needed > > > > > > > > > Assuming you'd stream to a file, you would need to fork off a curl with > > the > > > parameters to write to a file. See system/main/task.ijs and > fork_jtask_ . > > > You would then need J to continue and periodically read the file (in a > > loop > > > or use a timer). The file reading routine should probably remember the > > file > > > length the last time it read it and then seek to that position if the > > file > > > length has changed. You could then look for the delimiter as the other > > > examples do. That routine should also check to see if the forked curl > > > process is still alive and if not should probably restart it (this > could > > be > > > because the network faulted or got disconnected for some other reason). > > > > > > That's probably how I would approach it. Does this help? > > > > > > > > > [1] - http://developer.oanda.com/docs/v1/stream/ > > > [2] - > > > > > > > > > https://github.com/oanda/java-api-streaming/blob/master/src/main/java/JavaApiStreaming.java > > > , https://github.com/oanda/py-api-streaming/blob/master/streaming.py > > > [3] - > > > > > > > > > http://www.jsoftware.com/jwiki/Scripts/Socket%20System?action=AttachFile&do=view&target=webio.ijs > > > > > > > > > > > > > > > On Tue, Jul 15, 2014 at 1:00 AM, Francisco Gonzalez Velasco < > > > [email protected]> wrote: > > > > > > > Hello, > > > > > > > > Thanks for your answer. > > > > > > > > I need an event for every record which comes in, and yes, I'd need to > > see > > > > some code writing streaming data to file, triggering an event and > > > reacting > > > > to an event. > > > > > > > > Cheers > > > > > > > > Francisco > > > > > > > > > > > > 2014-07-15 6:43 GMT+02:00 Raul Miller <[email protected]>: > > > > > > > > > I think I'd stream to file, and have J busy wait (perhaps a second > of > > > > sleep > > > > > while waiting, to keep load down) when it's caught up. > > > > > > > > > > Would you want an event for every record which comes in, or would > it > > > make > > > > > more sense to skip forward to the most recent complete event when > > more > > > > data > > > > > becomes available? Or would you want to experiment with this? > > > > > > > > > > Also, is that enough of an explanation? I could draft something > up, I > > > > > guess. (But not exactly right now.) > > > > > > > > > > Thanks, > > > > > > > > > > -- > > > > > Raul > > > > > > > > > > > > > > > On Tue, Jul 15, 2014 at 12:14 AM, Francisco Gonzalez Velasco < > > > > > [email protected]> wrote: > > > > > > > > > > > Hello, my experience programming in J is very limited, so Inee > your > > > > > help... > > > > > > > > > > > > I´m using J64-602 to connect to a financial data source using > their > > > > REST > > > > > > API, so I can get static data: > > > > > > > > > > > > To get the information about EUR USD in the last 5 days > > > > > > (instrument=EUR_USD&granularity=D&count=5) I use this command: > > > > > > > > > > > > load 'web/gethttp' > > > > > > > load 'convert/json' > > > > > > > > > > > > >>(<1 2){dec_json('-s -k -H "Authorization: Bearer > > > > > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"') > > > > > > gethttp' > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://api-fxpractice.oanda.com/v1/candles?instrument=EUR_USD&granularity=D&count=5&candleFormat=bidask > > > > > > > ' > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > > > │time > > │openBid│openAsk│highBid│highAsk│lowBid > > > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > > > > > > > > > > > > > > > > > > > │2014-06-10T21:00:00.000000Z│1.35431│1.35472│1.35565│1.35576│1.35212│1.35224│1.353 > > > > > > > │1.3532 │42528 │json_true │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > > > │time > > │openBid│openAsk│highBid│highAsk│lowBid > > > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > │2014-06-11T21:00:00.000000Z│1.353 │1.35319│1.35715│1.3573 > > > > > > > │1.35117│1.35129│1.35494 │1.35536 │54799 │json_true │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > > > │time > > │openBid│openAsk│highBid│highAsk│lowBid > > > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > > > > > > > > > > > > > > > > > > > │2014-06-12T21:00:00.000000Z│1.35511│1.35566│1.35786│1.35799│1.35204│1.35215│1.35374 > > > > > > > │1.35468 │46632 │json_true │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > > > │time > > │openBid│openAsk│highBid│highAsk│lowBid > > > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > > > > > > > > > > > > > > > > > > > │2014-06-15T21:00:00.000000Z│1.35391│1.35458│1.35791│1.35804│1.35124│1.35137│1.35733 > > > > > > > │1.35747 │39482 │json_true │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > > > │time > > │openBid│openAsk│highBid│highAsk│lowBid > > > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > │2014-06-16T21:00:00.000000Z│1.35716│1.3573 > > > > > > > │1.35866│1.35878│1.35526│1.35538│1.35566 │1.35577 │10173 > > > │json_false│ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > > > > > > Now I need to know how to use streaming data... Data will be > > > delivered > > > > in > > > > > > real time. > > > > > > > > > > > > I would use last information arrived and store data to form a > > > database. > > > > > > > > > > > > The command to get the streaming data is: > > > > > > > > > > > > ('-s -k -H "Authorization: Bearer > > > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"') > > > > > > gethttp'https://stream-fxpractice.oanda.com > > > > > > /v1/prices?accountId=abcdefg&instruments=EUR_USD' > > > > > > > > > > > > > > > > > > When I use curl in the console to get streaming data the command > > is: > > > > > > > > > > > > C:\Users\Francisco>curl -k -H "Authorization: Bearer > > > > > > XXXXXXXXXXXXXXXXXXXXXXX > > > > > > > XXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" " > > > > > > > https://stream-fxpractice.oanda.com/v1/prices?accountId= > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://stream-fxpractice.oanda.com/v1/prices?accountId=1125870&instruments=EUR_USD > > > > > > > > > > > > > > abcdefg&instruments=EUR_USD > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://www.google.com/url?q=https%3A%2F%2Fstream-fxpractice.oanda.com%2Fv1%2Fprices%3FaccountId%3D1125870%26instruments%3DEUR_USD&sa=D&sntz=1&usg=AFQjCNEQy2PxdhBR4bYOLWuHEZSp37Xa-A > > > > > > > > > > > > > > " > > > > > > > > > > > > > > > > > > And the incoming data is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.313886Z","bid":1.35744,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.413276Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.423912Z","bid":1.35743,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.518162Z","bid":1.35744,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.618055Z","bid":1.35743,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.691201Z","bid":1.35744,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.824403Z","bid":1.35745,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:12.250563Z","bid":1.35745,"ask":1.35758} > > > > > > > {"heartbeat":{"time":"2014-06-17T08:50:12.547586Z"}} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:12.611163Z","bid":1.35746,"ask":1.35758} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:13.168306Z","bid":1.35746,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:13.527477Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.009987Z","bid":1.35746,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.109158Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.228386Z","bid":1.35743,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.314567Z","bid":1.35744,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.912960Z","bid":1.35742,"ask":1.35753} > > > > > > > {"heartbeat":{"time":"2014-06-17T08:50:15.036929Z"}} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:15.214964Z","bid":1.3574,"ask":1.35751} > > > > > > > > > > > > > > > > > > Token and account number have been changed > > > > > > > > > > > > > > > > > > 1) I'd like to know how to have a variable holding the most > recent > > > data > > > > > > sent by the data provider. And an event triggering when a new > value > > > > > arrives > > > > > > so actions can be taken depending on the values. > > > > > > 2) I'd like to write the incoming data to disk (as a csv file, > > > JDB,...) > > > > > > for persistance. > > > > > > > > > > > > Cheers > > > > > > > > > > > > Francisco > > > > > > > > > ---------------------------------------------------------------------- > > > > > > For information about J forums see > > > http://www.jsoftware.com/forums.htm > > > > > > > ---------------------------------------------------------------------- > > > > > For information about J forums see > > http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > > > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
