Klaus, let's go back to your original post where you posted this log:
09/30/12 23:59:05.196 IB API 2103: Market data farm connection is
broken:usfuture
09/30/12 23:59:21.450 IB API 2104: Market data farm connection is
OK:usfuture
10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please
empty deep book contents before applying any new entries. (for id 3)
10/01/12 00:14:04.896 My JBookTrader Market data for book 3 has been reset.
10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please
empty deep book contents before applying any new entries. (for id 2)
10/01/12 00:14:04.896 My JBookTrader Market data for book 2 has been reset.
10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET. Please
empty deep book contents before applying any new entries. (for id 1)
10/01/12 00:14:04.896 My JBookTrader Market data for book 1 has been reset.
10/01/12 00:14:04.896 IB API 2103: Market data farm connection is
broken:usfuture
10/01/12 00:14:42.166 IB API 2104: Market data farm connection is
OK:usfuture
10/01/12 00:15:48.860 IB API 1100: Connectivity between IB and Trader
Workstation has been lost.
10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and Trader
Workstation has been restored - data maintained.
Between 23:59:05 and 00:16:16, there is some activity indicating IB restart
processes kicking in, which ultimately end at 00:16:16. The message says
"Connectivity between IB and Trader Workstation has been restored - data
maintained", which means that all market data subscriptions are still
valid. But you are saying that the price data never changed after 00:16:16.
Correct?
On Tuesday, October 9, 2012 2:48:01 PM UTC-4, Klaus wrote:
>
> Dear Eugene, Dear Judson,
>
> first thanks to all for clarifying this. This was puzzling me and now I
> understand how it can technically
> happen. I was mistaken to believe it would be the same stream.
>
> Certainly, if you give me a proposed patch, I would be happy to analyze
> the relevant effects.
>
> Cheers
> Klaus
>
>
> Am Dienstag, 9. Oktober 2012 14:43:00 UTC-4 schrieb nonlinear:
>>
>> Correct, Judson.
>>
>> The book data is requested via this line in TraderAssistant.java:
>> socket.reqMktDepth(ticker, contract, 10);
>> The corresponding callback method in Trader.java:
>> updateMktDepth();
>> This is where the price is determined as the midpoint between the current
>> best bid and current best ask.
>>
>> The volume data is requested via this line in TraderAssistant.java:
>> socket.reqMktData(ticker, contract, "", false);
>> The corresponding callback method in Trader.java:
>> tickSize()
>> Price data (such as the "last price") can also be handled here, but JBT
>> only uses the volume data here.
>>
>> The subscription to market depth and volume happens here:
>> if (!subscribedTickers.contains(ticker)) {
>> subscribedTickers.add(ticker);
>> socket.reqContractDetails(strategy.getContract().m_conId,
>> strategy.getContract());
>> eventReport.report(JBookTrader.APP_NAME, "Requested contract
>> details for instrument " + instrument);
>> socket.reqMktDepth(ticker, contract, 10);
>> eventReport.report(JBookTrader.APP_NAME, "Requested book data
>> for instrument " + instrument);
>> socket.reqMktData(ticker, contract, "", false);
>> eventReport.report(JBookTrader.APP_NAME, "Requested market
>> data for instrument " + instrument);
>> }
>>
>>
>> What I suggested to Klaus is that we can detect the loss of market data
>> via the error() method, and resubscribe if this happens. Klaus, I'll see if
>> I can make that code change and I'll send it over to you to test. Are you
>> OK with it?
>>
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2012 at 1:58 PM, Judson Wilson <[email protected]>wrote:
>>
>>> I thought price was not coming from the same stream as volume, i
>>> thought it comes from the book stream (isn't it the midpoint of the
>>> best bid/ask on the book?)
>>>
>>>
>>> On Tue, Oct 9, 2012 at 8:59 AM, Klaus <[email protected]> wrote:
>>> > Dear nonlinear,
>>> >
>>> > what irritates me most is that it partially recovers: some data (for
>>> the
>>> > same instrument) is reconnected and correctly logged (volume = at
>>> least I
>>> > assume it is correct, it is at least reasonable), while price is not
>>> > correctly logged. To me this looks more like a SW issue, as both types
>>> of
>>> > data are coming from the same stream.
>>> > Regarding network connection, I assume there is probably few better
>>> > connections than AWS. It certainly is way ahead of private
>>> > connections I dealt with. It might also be interesting to note that I
>>> do
>>> > not use TWS, but IB Gateway.
>>> >
>>> > Cheers
>>> > Klaus
>>> >
>>> >
>>> > Am Montag, 8. Oktober 2012 10:44:27 UTC-4 schrieb nonlinear:
>>> >>
>>> >> Thanks for reporting the problem, Klaus. With respect to the midnight
>>> >> market data interruptions, this has always been the case, as IB
>>> servers
>>> >> restart at that time. The expected behavior is that once the restart
>>> is
>>> >> completed, the market data resumes and TWS and JBT would resume
>>> receiving
>>> >> the data stream without the need to restart. However, I noticed that
>>> >> occasionally this fails. My remedy is restart both TWS and JBT every
>>> day in
>>> >> the morning before regular trading starts. With respect to the
>>> interruptions
>>> >> during the other times of the day, I belive it all has to do with the
>>> >> quality of your network connection. The better quality, the less
>>> frequent is
>>> >> the loss. In my experience, both JBT and TWS recover from these
>>> >> interruptions (except for the occasional non-recoverable midnight
>>> resets).
>>> >> The typical pattern in the log is the message "market data lost",
>>> followed
>>> >> by in a minute or two by message "market data restored". Now, with
>>> respect
>>> >> to the stock data that you see in the TWS log, it's probably your
>>> standard
>>> >> subscription. I'd suggest visiting your "market subscriptions" page
>>> >> associated with your account on the IB we site.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Oct 8, 2012 at 9:45 AM, Klaus <[email protected]> wrote:
>>> >>>
>>> >>> Just had the same again. This time at 3:25 Monday morning.
>>> >>> Registered a temporary connection loss in the event report. Book and
>>> >>> price data did not change for any of the collected data from
>>> thereon, but
>>> >>> volume does.
>>> >>>
>>> >>> In addition, I found the following messages in the terminal window.
>>> >>> Perhaps somehow can make sense of all this?
>>> >>> (I include one message before and after for context)
>>> >>> ---
>>> >>> 03:08:11:547 JTS-CCPPing-269: Warning: not received heartbeat at
>>> 20904000
>>> >>> elapsed:11000
>>> >>> 03:25:30:728 JTS-PostAuthenticate-513: pre-logon message successfully
>>> >>> sent
>>> >>> 03:25:36:225 AWT-EventQueue-0: Warning: hot restart not equal to
>>> quick
>>> >>> restart
>>> >>> 03:25:36:226 AWT-EventQueue-0: Unsubscribe MD before routing table
>>> >>> rerequest
>>> >>> 03:25:36:227 AWT-EventQueue-0: Unsubscribe HMDS before updating
>>> routing
>>> >>> table
>>> >>> 03:25:36:227 AWT-EventQueue-0: Could not find route table entry for
>>> >>> path:exch=NEWS secType=STK currency=USD dataType=DayChart
>>> >>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available
>>> for
>>> >>> the exchange 'NEWS' and the security type 'Stock'
>>> >>> 03:25:36:228 AWT-EventQueue-0: Could not find route table entry for
>>> >>> path:exch=RTRSFND secType=STK currency=USD dataType=DayChart
>>> >>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available
>>> for
>>> >>> the exchange 'RTRSFND' and the security type 'Stock'
>>> >>> 03:25:36:228 AWT-EventQueue-0: Could not find route table entry for
>>> >>> path:exch=RTRSFND secType=STK currency=USD dataType=DayChart
>>> >>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is available
>>> for
>>> >>> the exchange 'RTRSFND' and the security type 'Stock'
>>> >>> 03:25:36:229 AWT-EventQueue-0: Reconnected!
>>> >>> 03:25:37:151 JTS-ForwardRouting-HISTORICAL_DATA-532:
>>> >>> handleCompatibleRouting HISTORICAL_DATA compmode=true ready=false
>>> >>> farm=ushmds
>>> >>> 03:25:37:939 JTS-ForwardRouting-MARKET_DATA-536:
>>> handleCompatibleRouting
>>> >>> MARKET_DATA compmode=true ready=false farm=usfarm
>>> >>> 03:26:36:399 JTS-Fuse-sd-533: Reset transferrable socket:ushmds
>>> >>> gw1.ibllc.com:4000
>>> >>> 03:26:37:187 JTS-Fuse-sd-537: Reset transferrable socket:usfarm
>>> >>> gw1.ibllc.com:4000
>>> >>> 05:20:18:052 JTS-CCPPing-516: Warning: not received heartbeat at
>>> 28813000
>>> >>> elapsed:11000
>>> >>> ------
>>> >>>
>>> >>> Data that is collected is only for the three instruments: EUR, NQ, ES
>>> >>> (thus I am puzzled also by the stock request above)
>>> >>>
>>> >>> Cheers
>>> >>> Klaus
>>> >>>
>>> >>>
>>> >>> Am Montag, 1. Oktober 2012 10:42:52 UTC-4 schrieb Klaus:
>>> >>>>
>>> >>>> Hi,
>>> >>>>
>>> >>>> for quite some time, I am using JBT to gather data. For this
>>> purpose, it
>>> >>>> is running on a AWS micro-instance.
>>> >>>> Once in a while I found it stuck (actually , but was not sure where
>>> it
>>> >>>> was coming from). Usually the happened also after a
>>> >>>> full week of data.
>>> >>>>
>>> >>>> However, today it happened again and I found some strange things (I
>>> >>>> start recording usually some time Sunday evening and let it run till
>>> >>>> Friday). Perhaps someone has an idea where it might come from. The
>>> fact is,
>>> >>>> it seems not data itself gets stuck only the price information is
>>> kept the
>>> >>>> same: the volume does continue to change.
>>> >>>>
>>> >>>> This is how the end of the recording looks like: then there is a
>>> 30sec
>>> >>>> gap where I restarted JBT.
>>> >>>>
>>> >>>> 100112,101425,-11.6,1.28335,14
>>> >>>>
>>> >>>> 100112,101426,-11.6,1.28335,6
>>> >>>>
>>> >>>> 100112,101427,-11.6,1.28335,0
>>> >>>>
>>> >>>> 100112,101458,9.95,1.29335,0
>>> >>>>
>>> >>>> 100112,101459,9.16,1.29335,2
>>> >>>>
>>> >>>> 100112,101500,8.24,1.29335,1
>>> >>>>
>>> >>>> The gap is not due to a change in the market, but because the data
>>> >>>> before was no longer correct (btw, this is the EUR)
>>> >>>>
>>> >>>>
>>> >>>> This is the part of the data where it seemed to happen:
>>> >>>>
>>> >>>> 100112,001556,-9.99,1.2833,0
>>> >>>>
>>> >>>> 100112,001557,-4.27,1.28325,0
>>> >>>>
>>> >>>> 100112,001558,-2.62,1.28325,0
>>> >>>>
>>> >>>> 100112,001559,-11.47,1.28335,16
>>> >>>>
>>> >>>> 100112,001600,-9.75,1.2833,1
>>> >>>>
>>> >>>> 100112,001601,-10.32,1.2833,0
>>> >>>>
>>> >>>> 100112,001602,-10.26,1.2833,0
>>> >>>>
>>> >>>> 100112,001603,-13.42,1.28335,1
>>> >>>>
>>> >>>> 100112,001604,-13.61,1.28335,0
>>> >>>>
>>> >>>> 100112,001605,-12.64,1.28335,0
>>> >>>>
>>> >>>> 100112,001606,-12.04,1.28335,0
>>> >>>>
>>> >>>> 100112,001607,-11.83,1.28335,0
>>> >>>>
>>> >>>> 100112,001608,-11.71,1.28335,0
>>> >>>>
>>> >>>> 100112,001609,-11.71,1.28335,0
>>> >>>>
>>> >>>> 100112,001610,-11.71,1.28335,0
>>> >>>>
>>> >>>> 100112,001611,-11.71,1.28335,0
>>> >>>>
>>> >>>> 100112,001612,-11.93,1.28335,0
>>> >>>>
>>> >>>> 100112,001613,-11.98,1.28335,0
>>> >>>>
>>> >>>> 100112,001614,-11.04,1.28335,7
>>> >>>>
>>> >>>> the price keeped changing until 001602, then was stuck - as far as I
>>> >>>> could see in the file all the way until I restarted.
>>> >>>>
>>> >>>>
>>> >>>> The following is an excerpt from the Eventreport file - it contains
>>> the
>>> >>>> end before the restart (the were no further events recorded)
>>> >>>>
>>> >>>> 09/30/12 23:59:05.196 IB API 2103: Market data farm connection is
>>> >>>> broken:usfuture
>>> >>>>
>>> >>>> 09/30/12 23:59:21.450 IB API 2104: Market data farm connection is
>>> >>>> OK:usfuture
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET.
>>> >>>> Please empty deep book contents before applying any new entries.
>>> (for id 3)
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 3 has been
>>> >>>> reset.
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET.
>>> >>>> Please empty deep book contents before applying any new entries.
>>> (for id 2)
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 2 has been
>>> >>>> reset.
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET.
>>> >>>> Please empty deep book contents before applying any new entries.
>>> (for id 1)
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 1 has been
>>> >>>> reset.
>>> >>>>
>>> >>>> 10/01/12 00:14:04.896 IB API 2103: Market data farm connection is
>>> >>>> broken:usfuture
>>> >>>>
>>> >>>> 10/01/12 00:14:42.166 IB API 2104: Market data farm connection is
>>> >>>> OK:usfuture
>>> >>>>
>>> >>>> 10/01/12 00:15:48.860 IB API 1100: Connectivity between IB and
>>> Trader
>>> >>>> Workstation has been lost.
>>> >>>>
>>> >>>> 10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and
>>> Trader
>>> >>>> Workstation has been restored - data maintained.
>>> >>>>
>>> >>>>
>>> >>>> So, it looks like connectivity was lost and restored, but from this
>>> >>>> point forward no price updates were recorded..
>>> >>>>
>>> >>>> Any idea what is going on, would be helpful.
>>> >>>>
>>> >>>> Cheers
>>> >>>> Klaus
>>> >>>>
>>> >>>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> Groups
>>> >>> "JBookTrader" group.
>>> >>> To view this discussion on the web visit
>>> >>> https://groups.google.com/d/msg/jbooktrader/-/O4d4rFkJ3F0J.
>>> >>>
>>> >>> To post to this group, send email to [email protected].
>>> >>> To unsubscribe from this group, send email to
>>> >>> [email protected].
>>> >>>
>>> >>> For more options, visit this group at
>>> >>> http://groups.google.com/group/jbooktrader?hl=en.
>>> >>
>>> >>
>>> >>
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "JBookTrader" group.
>>> > To view this discussion on the web visit
>>> > https://groups.google.com/d/msg/jbooktrader/-/tEYk3_95L5EJ.
>>> >
>>> > To post to this group, send email to [email protected].
>>> > To unsubscribe from this group, send email to
>>> > [email protected].
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/jbooktrader?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "JBookTrader" group.
>>> To post to this group, send email to [email protected].
>>> To unsubscribe from this group, send email to
>>> [email protected].
>>> For more options, visit this group at
>>> http://groups.google.com/group/jbooktrader?hl=en.
>>>
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"JBookTrader" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/jbooktrader/-/tmhQOD8_NU0J.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/jbooktrader?hl=en.
