Dear Eugene I usually do not use TWS for two reasons: it needs more memory and while it currently is not a problem, TWS is simply heavier. Further, TWS auto-logsout after 24h and while there are ways around it, they are all rather messy. If you say, it is just a few lines of code: if you point me to the critical aspects, I can probably fix it on my version, dry-run it for some weeks and if problems seem solved, this would just go in the regular version. How does this sound?
Cheers Klaus Am Dienstag, 9. Oktober 2012 12:57:57 UTC-4 schrieb nonlinear: > > One thing that may be worth trying is for JBT to detect when the price > feed stops, and when this happens, unsubscribe from the market data and > subscribe again. This should be just a few lines of code. Klaus, have you > tried pairing both TWS and IBGateway with JBT? Does it affect the frequency > and the severity of the market data interruptions? Like BJ, I've been > trading using JBT with IBGateway, and as I mentioned before, I don't see > any unrecoverable interruptions in the market data feed. Then again, I > restart IBGateway and JBT every morning, which I don't see as a big deal, > but I also ackwnoledge that it would be nice to be running continuously. > Actually, you do have to restart once in a while, simply because the > future's contract expires, and there is no auto-rolling in JBT. > > > > On Tue, Oct 9, 2012 at 12:14 PM, Klaus <[email protected] <javascript:> > > wrote: > >> Dear BJ >> >> thanks for reacting and reflecting on this. >> I am not sure what environment you are using. I admit, your description >> is not fully clear to me (e.g., the part of JBT sending to two differnt >> accounts - this probably implies you are using an adapted version as this >> is no standard JBT behavior). >> I use Ubuntu, on top of this I even use the full window interface and I >> then start JBT and IB-Gateway in it. >> What I then have is the following: (data with some additional shells open >> and all this) >> 579072 (out of 609480K) used, 31400K free, 5312 buffers and 0 swap usage >> (103824K cached) >> This looks a pretty comfortable setup to me, memory-wise. >> Nevertheless in a VM environment there might occur issues like once the >> VM is migrated, this might lead to an interruption of the >> data connection (but this can also happen with larger VMs of course). >> So, I am not concerned by the interruption per se. This actually happens >> several times a week. As nonlinear wrote: this is s.th. JBT is supposed >> to deal with. The problem here is that it recovers the connection, but >> somehow not fully, which would be difficult to explain from a connection or >> hardware side, but seems to me (without deep JBT-knowledge) more like a >> SW-issue. >> >> Of course, I can now try it in a standard VM. - probably I will for next >> week. But actually this is does not prove much. This runs for 10months now >> in the microinstance, including all updates and everything. Once in a while >> I get disconnects, but it was only by chance (I assume) that it now >> happened in two consecutive weeks. So, if there is no problem with a >> standard instance, this does not mean it avoided a problem that would have >> happened with a micro, as this is ok most of the time as well. >> I actually also did not think, about it stalling to much. Glitches >> happen. - What made it so surprising to me is that some data >> for the Future is still correctly tracked, while other not at all: this >> seems very weird to me.. - and I am looking for a good hypothesis to >> understand this. >> >> Cheers >> Klaus >> >> >> >> >> >> Am Montag, 8. Oktober 2012 14:46:31 UTC-4 schrieb bj: >> >>> I'm running JBT on an AWS Small instance and tried switching to a Micro >>> instance over the weekend. I use IB Gateway instead of TWS (it doesn't >>> shutdown itself and uses less memory). After starting IB Gateway on the >>> micro instance it used all the memory. I started another IB Gateway >>> instance (my JBT sends orders to two different IB accounts). The result >>> was, that while Ubuntu was freeing the memory for the second instance, the >>> first one was shutdown by the OS. >>> >>> I then started only one IB Gateway and JBT and monitored the memory. >>> Over the weekend without any market data processing the memory was always >>> 99% full. I didn't feel comfortable running JBT in such a low memory >>> environment so I switched back to a small instance. >>> >>> What I'm trying to say is switch to a small instance for a week and see >>> if that solves your problems. >>> >>> >>> On Monday, October 8, 2012 4:44:27 PM UTC+2, nonlinear wrote: >>>> >>>> Thanks for reporting the problem, Klaus. With respect to the midnight >>>> market data interruptions, this has always been the case, as IB servers >>>> restart at that time. The expected behavior is that once the restart is >>>> completed, the market data resumes and TWS and JBT would resume receiving >>>> the data stream without the need to restart. However, I noticed that >>>> occasionally this fails. My remedy is restart both TWS and JBT every day >>>> in >>>> the morning before regular trading starts. With respect to the >>>> interruptions during the other times of the day, I belive it all has to do >>>> with the quality of your network connection. The better quality, the less >>>> frequent is the loss. In my experience, both JBT and TWS recover from >>>> these >>>> interruptions (except for the occasional non-recoverable midnight resets). >>>> The typical pattern in the log is the message "market data lost", followed >>>> by in a minute or two by message "market data restored". Now, with respect >>>> to the stock data that you see in the TWS log, it's probably your standard >>>> subscription. I'd suggest visiting your "market subscriptions" page >>>> associated with your account on the IB we site. >>>> >>>> >>>> >>>> >>>> On Mon, Oct 8, 2012 at 9:45 AM, Klaus <[email protected]> wrote: >>>> >>>>> Just had the same again. This time at 3:25 Monday morning. >>>>> Registered a temporary connection loss in the event report. Book and >>>>> price data did not change for any of the collected data from thereon, but >>>>> volume does. >>>>> >>>>> In addition, I found the following messages in the terminal window. >>>>> Perhaps somehow can make sense of all this? >>>>> (I include one message before and after for context) >>>>> --- >>>>> 03:08:11:547 JTS-CCPPing-269: Warning: not received heartbeat at >>>>> 20904000 elapsed:11000 >>>>> 03:25:30:728 JTS-PostAuthenticate-513: pre-logon message successfully >>>>> sent >>>>> 03:25:36:225 AWT-EventQueue-0: Warning: hot restart not equal to quick >>>>> restart >>>>> 03:25:36:226 AWT-EventQueue-0: Unsubscribe MD before routing table >>>>> rerequest >>>>> 03:25:36:227 AWT-EventQueue-0: Unsubscribe HMDS before updating >>>>> routing table >>>>> 03:25:36:227 AWT-EventQueue-0: Could not find route table entry for >>>>> path:exch=NEWS secType=STK currency=USD dataType=DayChart >>>>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available >>>>> for the exchange 'NEWS' and the security type 'Stock' >>>>> 03:25:36:228 AWT-EventQueue-0: Could not find route table entry for >>>>> path:exch=RTRSFND secType=STK currency=USD dataType=DayChart >>>>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available >>>>> for the exchange 'RTRSFND' and the security type 'Stock' >>>>> 03:25:36:228 AWT-EventQueue-0: Could not find route table entry for >>>>> path:exch=RTRSFND secType=STK currency=USD dataType=DayChart >>>>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available >>>>> for the exchange 'RTRSFND' and the security type 'Stock' >>>>> 03:25:36:229 AWT-EventQueue-0: Reconnected! >>>>> 03:25:37:151 JTS-ForwardRouting-HISTORICAL_**DATA-532: >>>>> handleCompatibleRouting HISTORICAL_DATA compmode=true ready=false >>>>> farm=ushmds >>>>> 03:25:37:939 JTS-ForwardRouting-MARKET_**DATA-536: >>>>> handleCompatibleRouting MARKET_DATA compmode=true ready=false farm=usfarm >>>>> 03:26:36:399 JTS-Fuse-sd-533: Reset transferrable socket:ushmds >>>>> gw1.ibllc.com:4000 >>>>> 03:26:37:187 JTS-Fuse-sd-537: Reset transferrable socket:usfarm >>>>> gw1.ibllc.com:4000 >>>>> 05:20:18:052 JTS-CCPPing-516: Warning: not received heartbeat at >>>>> 28813000 elapsed:11000 >>>>> ------ >>>>> >>>>> Data that is collected is only for the three instruments: EUR, NQ, ES >>>>> (thus I am puzzled also by the stock request above) >>>>> >>>>> Cheers >>>>> Klaus >>>>> >>>>> >>>>> Am Montag, 1. Oktober 2012 10:42:52 UTC-4 schrieb Klaus: >>>>> >>>>>> Hi, >>>>>> >>>>>> for quite some time, I am using JBT to gather data. For this purpose, >>>>>> it is running on a AWS micro-instance. >>>>>> Once in a while I found it stuck (actually , but was not sure where >>>>>> it was coming from). Usually the happened also after a >>>>>> full week of data. >>>>>> >>>>>> However, today it happened again and I found some strange things (I >>>>>> start recording usually some time Sunday evening and let it run till >>>>>> Friday). Perhaps someone has an idea where it might come from. The fact >>>>>> is, >>>>>> it seems not data itself gets stuck only the price information is kept >>>>>> the >>>>>> same: the volume does continue to change. >>>>>> >>>>>> This is how the end of the recording looks like: then there is a >>>>>> 30sec gap where I restarted JBT. >>>>>> >>>>>> 100112,101425,-11.6,1.28335,14 >>>>>> >>>>>> 100112,101426,-11.6,1.28335,6 >>>>>> >>>>>> 100112,101427,-11.6,1.28335,0 >>>>>> >>>>>> 100112,101458,9.95,1.29335,0 >>>>>> >>>>>> 100112,101459,9.16,1.29335,2 >>>>>> >>>>>> 100112,101500,8.24,1.29335,1 >>>>>> >>>>>> The gap is not due to a change in the market, but because the data >>>>>> before was no longer correct (btw, this is the EUR) >>>>>> >>>>>> >>>>>> This is the part of the data where it seemed to happen: >>>>>> >>>>>> 100112,001556,-9.99,1.2833,0 >>>>>> >>>>>> 100112,001557,-4.27,1.28325,0 >>>>>> >>>>>> 100112,001558,-2.62,1.28325,0 >>>>>> >>>>>> 100112,001559,-11.47,1.28335,**1**6 >>>>>> >>>>>> 100112,001600,-9.75,1.2833,1 >>>>>> >>>>>> 100112,001601,-10.32,1.2833,0 >>>>>> >>>>>> 100112,001602,-10.26,1.2833,0 >>>>>> >>>>>> 100112,001603,-13.42,1.28335,1 >>>>>> >>>>>> 100112,001604,-13.61,1.28335,0 >>>>>> >>>>>> 100112,001605,-12.64,1.28335,0 >>>>>> >>>>>> 100112,001606,-12.04,1.28335,0 >>>>>> >>>>>> 100112,001607,-11.83,1.28335,0 >>>>>> >>>>>> 100112,001608,-11.71,1.28335,0 >>>>>> >>>>>> 100112,001609,-11.71,1.28335,0 >>>>>> >>>>>> 100112,001610,-11.71,1.28335,0 >>>>>> >>>>>> 100112,001611,-11.71,1.28335,0 >>>>>> >>>>>> 100112,001612,-11.93,1.28335,0 >>>>>> >>>>>> 100112,001613,-11.98,1.28335,0 >>>>>> >>>>>> 100112,001614,-11.04,1.28335,7 >>>>>> >>>>>> the price keeped changing until 001602, then was stuck - as far as I >>>>>> could see in the file all the way until I restarted. >>>>>> >>>>>> >>>>>> The following is an excerpt from the Eventreport file - it contains >>>>>> the end before the restart (the were no further events recorded) >>>>>> >>>>>> 09/30/12 23:59:05.196 IB API 2103: Market data farm connection is >>>>>> broken:usfuture >>>>>> >>>>>> 09/30/12 23:59:21.450 IB API 2104: Market data farm connection is >>>>>> OK:usfuture >>>>>> >>>>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. >>>>>> Please empty deep book contents before applying any new entries. (for id >>>>>> 3) >>>>>> >>>>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 3 has been >>>>>> reset. >>>>>> >>>>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. >>>>>> Please empty deep book contents before applying any new entries. (for id >>>>>> 2) >>>>>> >>>>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 2 has been >>>>>> reset. >>>>>> >>>>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. >>>>>> Please empty deep book contents before applying any new entries. (for id >>>>>> 1) >>>>>> >>>>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 1 has been >>>>>> reset. >>>>>> >>>>>> 10/01/12 00:14:04.896 IB API 2103: Market data farm connection is >>>>>> broken:usfuture >>>>>> >>>>>> 10/01/12 00:14:42.166 IB API 2104: Market data farm connection is >>>>>> OK:usfuture >>>>>> >>>>>> 10/01/12 00:15:48.860 IB API 1100: Connectivity between IB and >>>>>> Trader Workstation has been lost. >>>>>> >>>>>> 10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and >>>>>> Trader Workstation has been restored - data maintained. >>>>>> >>>>>> So, it looks like connectivity was lost and restored, but from this >>>>>> point forward no price updates were recorded.. >>>>>> >>>>>> Any idea what is going on, would be helpful. >>>>>> >>>>>> Cheers >>>>>> Klaus >>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "JBookTrader" group. >>>>> To view this discussion on the web visit https://groups.google.com/d/* >>>>> *msg/jbooktrader/-/O4d4rFkJ3F0J<https://groups.google.com/d/msg/jbooktrader/-/O4d4rFkJ3F0J> >>>>> **. >>>>> >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> jbooktrader...@googlegroups.**com. >>>>> For more options, visit this group at http://groups.google.com/** >>>>> group/jbooktrader?hl=en<http://groups.google.com/group/jbooktrader?hl=en> >>>>> . >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "JBookTrader" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/jbooktrader/-/JlHTQ1N14wAJ. >> >> To post to this group, send email to [email protected]<javascript:> >> . >> To unsubscribe from this group, send email to >> [email protected] <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/jbooktrader?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "JBookTrader" group. To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/CeL0hrGcd0QJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.
