Thanks for your help, I think this is far beyond my current abilities with J.
Cheers 2014-07-15 13:34 GMT+02:00 Joe Bogner <[email protected]>: > I researched this a bit and here is what I found: > > 1. oanda appears to use chunked encoding for streamed requests[1]. Curl > will automatically handle this > 2. There are several examples[2] available in other languages to learn from > 3. The examples will trigger a new record based on the presence of the word > instrument in the stream as a delimiter > 4. J's web/gethttp will use curl or wget and will handle chunked encoding > as a result, but I suspect it will wait for the connection to close before > returning any of the message > > The key decision to make in my opinion is whether to stream to a file or to > parse everything in memory > > Streaming to a file likely makes sense in this scenario. There can be > streaming use cases where it's impractical to stream to a file because the > stream would consume all disk space -- then you'd have to come up with some > rolling file mechanism, which would not easily be handled by curl/wget as > far as I know. > > Parsing in memory would require writing your own HTTP get utility. You > could take a look at Henry Rich's webio[3] which implements much of what > would be needed > > > Assuming you'd stream to a file, you would need to fork off a curl with the > parameters to write to a file. See system/main/task.ijs and fork_jtask_ . > You would then need J to continue and periodically read the file (in a loop > or use a timer). The file reading routine should probably remember the file > length the last time it read it and then seek to that position if the file > length has changed. You could then look for the delimiter as the other > examples do. That routine should also check to see if the forked curl > process is still alive and if not should probably restart it (this could be > because the network faulted or got disconnected for some other reason). > > That's probably how I would approach it. Does this help? > > > [1] - http://developer.oanda.com/docs/v1/stream/ > [2] - > > https://github.com/oanda/java-api-streaming/blob/master/src/main/java/JavaApiStreaming.java > , https://github.com/oanda/py-api-streaming/blob/master/streaming.py > [3] - > > http://www.jsoftware.com/jwiki/Scripts/Socket%20System?action=AttachFile&do=view&target=webio.ijs > > > > > On Tue, Jul 15, 2014 at 1:00 AM, Francisco Gonzalez Velasco < > [email protected]> wrote: > > > Hello, > > > > Thanks for your answer. > > > > I need an event for every record which comes in, and yes, I'd need to see > > some code writing streaming data to file, triggering an event and > reacting > > to an event. > > > > Cheers > > > > Francisco > > > > > > 2014-07-15 6:43 GMT+02:00 Raul Miller <[email protected]>: > > > > > I think I'd stream to file, and have J busy wait (perhaps a second of > > sleep > > > while waiting, to keep load down) when it's caught up. > > > > > > Would you want an event for every record which comes in, or would it > make > > > more sense to skip forward to the most recent complete event when more > > data > > > becomes available? Or would you want to experiment with this? > > > > > > Also, is that enough of an explanation? I could draft something up, I > > > guess. (But not exactly right now.) > > > > > > Thanks, > > > > > > -- > > > Raul > > > > > > > > > On Tue, Jul 15, 2014 at 12:14 AM, Francisco Gonzalez Velasco < > > > [email protected]> wrote: > > > > > > > Hello, my experience programming in J is very limited, so Inee your > > > help... > > > > > > > > I´m using J64-602 to connect to a financial data source using their > > REST > > > > API, so I can get static data: > > > > > > > > To get the information about EUR USD in the last 5 days > > > > (instrument=EUR_USD&granularity=D&count=5) I use this command: > > > > > > > > load 'web/gethttp' > > > > > load 'convert/json' > > > > > > > > >>(<1 2){dec_json('-s -k -H "Authorization: Bearer > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"') > > > > gethttp' > > > > > > > > > > > > > > > https://api-fxpractice.oanda.com/v1/candles?instrument=EUR_USD&granularity=D&count=5&candleFormat=bidask > > > > > ' > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > > > > > > │2014-06-10T21:00:00.000000Z│1.35431│1.35472│1.35565│1.35576│1.35212│1.35224│1.353 > > > > > │1.3532 │42528 │json_true │ > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > │2014-06-11T21:00:00.000000Z│1.353 │1.35319│1.35715│1.3573 > > > > > │1.35117│1.35129│1.35494 │1.35536 │54799 │json_true │ > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > > > > > > │2014-06-12T21:00:00.000000Z│1.35511│1.35566│1.35786│1.35799│1.35204│1.35215│1.35374 > > > > > │1.35468 │46632 │json_true │ > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > > > > > > │2014-06-15T21:00:00.000000Z│1.35391│1.35458│1.35791│1.35804│1.35124│1.35137│1.35733 > > > > > │1.35747 │39482 │json_true │ > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > │2014-06-16T21:00:00.000000Z│1.35716│1.3573 > > > > > │1.35866│1.35878│1.35526│1.35538│1.35566 │1.35577 │10173 > │json_false│ > > > > > > > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > > > > Now I need to know how to use streaming data... Data will be > delivered > > in > > > > real time. > > > > > > > > I would use last information arrived and store data to form a > database. > > > > > > > > The command to get the streaming data is: > > > > > > > > ('-s -k -H "Authorization: Bearer > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"') > > > > gethttp'https://stream-fxpractice.oanda.com > > > > /v1/prices?accountId=abcdefg&instruments=EUR_USD' > > > > > > > > > > > > When I use curl in the console to get streaming data the command is: > > > > > > > > C:\Users\Francisco>curl -k -H "Authorization: Bearer > > > > XXXXXXXXXXXXXXXXXXXXXXX > > > > > XXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" " > > > > > https://stream-fxpractice.oanda.com/v1/prices?accountId= > > > > > < > > > > > > > > > > https://stream-fxpractice.oanda.com/v1/prices?accountId=1125870&instruments=EUR_USD > > > > > > > > > > abcdefg&instruments=EUR_USD > > > > > < > > > > > > > > > > https://www.google.com/url?q=https%3A%2F%2Fstream-fxpractice.oanda.com%2Fv1%2Fprices%3FaccountId%3D1125870%26instruments%3DEUR_USD&sa=D&sntz=1&usg=AFQjCNEQy2PxdhBR4bYOLWuHEZSp37Xa-A > > > > > > > > > > " > > > > > > > > > > > > And the incoming data is: > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.313886Z","bid":1.35744,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.413276Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.423912Z","bid":1.35743,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.518162Z","bid":1.35744,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.618055Z","bid":1.35743,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.691201Z","bid":1.35744,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.824403Z","bid":1.35745,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:12.250563Z","bid":1.35745,"ask":1.35758} > > > > > {"heartbeat":{"time":"2014-06-17T08:50:12.547586Z"}} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:12.611163Z","bid":1.35746,"ask":1.35758} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:13.168306Z","bid":1.35746,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:13.527477Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.009987Z","bid":1.35746,"ask":1.35756} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.109158Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.228386Z","bid":1.35743,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.314567Z","bid":1.35744,"ask":1.35755} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.912960Z","bid":1.35742,"ask":1.35753} > > > > > {"heartbeat":{"time":"2014-06-17T08:50:15.036929Z"}} > > > > > > > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:15.214964Z","bid":1.3574,"ask":1.35751} > > > > > > > > > > > > Token and account number have been changed > > > > > > > > > > > > 1) I'd like to know how to have a variable holding the most recent > data > > > > sent by the data provider. And an event triggering when a new value > > > arrives > > > > so actions can be taken depending on the values. > > > > 2) I'd like to write the incoming data to disk (as a csv file, > JDB,...) > > > > for persistance. > > > > > > > > Cheers > > > > > > > > Francisco > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
