Dear Eugene,
right. I did not check every line, but it was exactly the same price at the
end and all checks in between gave the same data
(It had run by this time for 10 hours). It was only afterwards that I found
that exactly at the same time these events happened.
Klaus
> Between 23:59:05 and 00:16:16, there is some activity indicating IB
> restart processes kicking in, which ultimately end at 00:16:16. The message
> says "Connectivity between IB and Trader Workstation has been restored -
> data maintained", which means that all market data subscriptions are still
> valid. But you are saying that the price data never changed after 00:16:16.
> Correct?
>
>
>
> On Tuesday, October 9, 2012 2:48:01 PM UTC-4, Klaus wrote:
>>
>> Dear Eugene, Dear Judson,
>>
>> first thanks to all for clarifying this. This was puzzling me and now I
>> understand how it can technically
>> happen. I was mistaken to believe it would be the same stream.
>>
>> Certainly, if you give me a proposed patch, I would be happy to analyze
>> the relevant effects.
>>
>> Cheers
>> Klaus
>>
>>
>> Am Dienstag, 9. Oktober 2012 14:43:00 UTC-4 schrieb nonlinear:
>>>
>>> Correct, Judson.
>>>
>>> The book data is requested via this line in TraderAssistant.java:
>>> socket.reqMktDepth(ticker, contract, 10);
>>> The corresponding callback method in Trader.java:
>>> updateMktDepth();
>>> This is where the price is determined as the midpoint between the
>>> current best bid and current best ask.
>>>
>>> The volume data is requested via this line in TraderAssistant.java:
>>> socket.reqMktData(ticker, contract, "", false);
>>> The corresponding callback method in Trader.java:
>>> tickSize()
>>> Price data (such as the "last price") can also be handled here, but JBT
>>> only uses the volume data here.
>>>
>>> The subscription to market depth and volume happens here:
>>> if (!subscribedTickers.contains(ticker)) {
>>> subscribedTickers.add(ticker);
>>> socket.reqContractDetails(strategy.getContract().m_conId,
>>> strategy.getContract());
>>> eventReport.report(JBookTrader.APP_NAME, "Requested contract
>>> details for instrument " + instrument);
>>> socket.reqMktDepth(ticker, contract, 10);
>>> eventReport.report(JBookTrader.APP_NAME, "Requested book
>>> data for instrument " + instrument);
>>> socket.reqMktData(ticker, contract, "", false);
>>> eventReport.report(JBookTrader.APP_NAME, "Requested market
>>> data for instrument " + instrument);
>>> }
>>>
>>>
>>> What I suggested to Klaus is that we can detect the loss of market data
>>> via the error() method, and resubscribe if this happens. Klaus, I'll see if
>>> I can make that code change and I'll send it over to you to test. Are you
>>> OK with it?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Oct 9, 2012 at 1:58 PM, Judson Wilson <[email protected]>wrote:
>>>
>>>> I thought price was not coming from the same stream as volume, i
>>>> thought it comes from the book stream (isn't it the midpoint of the
>>>> best bid/ask on the book?)
>>>>
>>>>
>>>> On Tue, Oct 9, 2012 at 8:59 AM, Klaus <[email protected]> wrote:
>>>> > Dear nonlinear,
>>>> >
>>>> > what irritates me most is that it partially recovers: some data (for
>>>> the
>>>> > same instrument) is reconnected and correctly logged (volume = at
>>>> least I
>>>> > assume it is correct, it is at least reasonable), while price is not
>>>> > correctly logged. To me this looks more like a SW issue, as both
>>>> types of
>>>> > data are coming from the same stream.
>>>> > Regarding network connection, I assume there is probably few better
>>>> > connections than AWS. It certainly is way ahead of private
>>>> > connections I dealt with. It might also be interesting to note that
>>>> I do
>>>> > not use TWS, but IB Gateway.
>>>> >
>>>> > Cheers
>>>> > Klaus
>>>> >
>>>> >
>>>> > Am Montag, 8. Oktober 2012 10:44:27 UTC-4 schrieb nonlinear:
>>>> >>
>>>> >> Thanks for reporting the problem, Klaus. With respect to the midnight
>>>> >> market data interruptions, this has always been the case, as IB
>>>> servers
>>>> >> restart at that time. The expected behavior is that once the restart
>>>> is
>>>> >> completed, the market data resumes and TWS and JBT would resume
>>>> receiving
>>>> >> the data stream without the need to restart. However, I noticed that
>>>> >> occasionally this fails. My remedy is restart both TWS and JBT every
>>>> day in
>>>> >> the morning before regular trading starts. With respect to the
>>>> interruptions
>>>> >> during the other times of the day, I belive it all has to do with the
>>>> >> quality of your network connection. The better quality, the less
>>>> frequent is
>>>> >> the loss. In my experience, both JBT and TWS recover from these
>>>> >> interruptions (except for the occasional non-recoverable midnight
>>>> resets).
>>>> >> The typical pattern in the log is the message "market data lost",
>>>> followed
>>>> >> by in a minute or two by message "market data restored". Now, with
>>>> respect
>>>> >> to the stock data that you see in the TWS log, it's probably your
>>>> standard
>>>> >> subscription. I'd suggest visiting your "market subscriptions" page
>>>> >> associated with your account on the IB we site.
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Mon, Oct 8, 2012 at 9:45 AM, Klaus <[email protected]> wrote:
>>>> >>>
>>>> >>> Just had the same again. This time at 3:25 Monday morning.
>>>> >>> Registered a temporary connection loss in the event report. Book and
>>>> >>> price data did not change for any of the collected data from
>>>> thereon, but
>>>> >>> volume does.
>>>> >>>
>>>> >>> In addition, I found the following messages in the terminal window.
>>>> >>> Perhaps somehow can make sense of all this?
>>>> >>> (I include one message before and after for context)
>>>> >>> ---
>>>> >>> 03:08:11:547 JTS-CCPPing-269: Warning: not received heartbeat at
>>>> 20904000
>>>> >>> elapsed:11000
>>>> >>> 03:25:30:728 JTS-PostAuthenticate-513: pre-logon message
>>>> successfully
>>>> >>> sent
>>>> >>> 03:25:36:225 AWT-EventQueue-0: Warning: hot restart not equal to
>>>> quick
>>>> >>> restart
>>>> >>> 03:25:36:226 AWT-EventQueue-0: Unsubscribe MD before routing table
>>>> >>> rerequest
>>>> >>> 03:25:36:227 AWT-EventQueue-0: Unsubscribe HMDS before updating
>>>> routing
>>>> >>> table
>>>> >>> 03:25:36:227 AWT-EventQueue-0: Could not find route table entry for
>>>> >>> path:exch=NEWS secType=STK currency=USD dataType=DayChart
>>>> >>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is
>>>> available for
>>>> >>> the exchange 'NEWS' and the security type 'Stock'
>>>> >>> 03:25:36:228 AWT-EventQueue-0: Could not find route table entry for
>>>> >>> path:exch=RTRSFND secType=STK currency=USD dataType=DayChart
>>>> >>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is
>>>> available for
>>>> >>> the exchange 'RTRSFND' and the security type 'Stock'
>>>> >>> 03:25:36:228 AWT-EventQueue-0: Could not find route table entry for
>>>> >>> path:exch=RTRSFND secType=STK currency=USD dataType=DayChart
>>>> >>> 03:25:36:228 AWT-EventQueue-0: No data of type DayChart is
>>>> available for
>>>> >>> the exchange 'RTRSFND' and the security type 'Stock'
>>>> >>> 03:25:36:229 AWT-EventQueue-0: Reconnected!
>>>> >>> 03:25:37:151 JTS-ForwardRouting-HISTORICAL_DATA-532:
>>>> >>> handleCompatibleRouting HISTORICAL_DATA compmode=true ready=false
>>>> >>> farm=ushmds
>>>> >>> 03:25:37:939 JTS-ForwardRouting-MARKET_DATA-536:
>>>> handleCompatibleRouting
>>>> >>> MARKET_DATA compmode=true ready=false farm=usfarm
>>>> >>> 03:26:36:399 JTS-Fuse-sd-533: Reset transferrable socket:ushmds
>>>> >>> gw1.ibllc.com:4000
>>>> >>> 03:26:37:187 JTS-Fuse-sd-537: Reset transferrable socket:usfarm
>>>> >>> gw1.ibllc.com:4000
>>>> >>> 05:20:18:052 JTS-CCPPing-516: Warning: not received heartbeat at
>>>> 28813000
>>>> >>> elapsed:11000
>>>> >>> ------
>>>> >>>
>>>> >>> Data that is collected is only for the three instruments: EUR, NQ,
>>>> ES
>>>> >>> (thus I am puzzled also by the stock request above)
>>>> >>>
>>>> >>> Cheers
>>>> >>> Klaus
>>>> >>>
>>>> >>>
>>>> >>> Am Montag, 1. Oktober 2012 10:42:52 UTC-4 schrieb Klaus:
>>>> >>>>
>>>> >>>> Hi,
>>>> >>>>
>>>> >>>> for quite some time, I am using JBT to gather data. For this
>>>> purpose, it
>>>> >>>> is running on a AWS micro-instance.
>>>> >>>> Once in a while I found it stuck (actually , but was not sure
>>>> where it
>>>> >>>> was coming from). Usually the happened also after a
>>>> >>>> full week of data.
>>>> >>>>
>>>> >>>> However, today it happened again and I found some strange things (I
>>>> >>>> start recording usually some time Sunday evening and let it run
>>>> till
>>>> >>>> Friday). Perhaps someone has an idea where it might come from. The
>>>> fact is,
>>>> >>>> it seems not data itself gets stuck only the price information is
>>>> kept the
>>>> >>>> same: the volume does continue to change.
>>>> >>>>
>>>> >>>> This is how the end of the recording looks like: then there is a
>>>> 30sec
>>>> >>>> gap where I restarted JBT.
>>>> >>>>
>>>> >>>> 100112,101425,-11.6,1.28335,14
>>>> >>>>
>>>> >>>> 100112,101426,-11.6,1.28335,6
>>>> >>>>
>>>> >>>> 100112,101427,-11.6,1.28335,0
>>>> >>>>
>>>> >>>> 100112,101458,9.95,1.29335,0
>>>> >>>>
>>>> >>>> 100112,101459,9.16,1.29335,2
>>>> >>>>
>>>> >>>> 100112,101500,8.24,1.29335,1
>>>> >>>>
>>>> >>>> The gap is not due to a change in the market, but because the data
>>>> >>>> before was no longer correct (btw, this is the EUR)
>>>> >>>>
>>>> >>>>
>>>> >>>> This is the part of the data where it seemed to happen:
>>>> >>>>
>>>> >>>> 100112,001556,-9.99,1.2833,0
>>>> >>>>
>>>> >>>> 100112,001557,-4.27,1.28325,0
>>>> >>>>
>>>> >>>> 100112,001558,-2.62,1.28325,0
>>>> >>>>
>>>> >>>> 100112,001559,-11.47,1.28335,16
>>>> >>>>
>>>> >>>> 100112,001600,-9.75,1.2833,1
>>>> >>>>
>>>> >>>> 100112,001601,-10.32,1.2833,0
>>>> >>>>
>>>> >>>> 100112,001602,-10.26,1.2833,0
>>>> >>>>
>>>> >>>> 100112,001603,-13.42,1.28335,1
>>>> >>>>
>>>> >>>> 100112,001604,-13.61,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001605,-12.64,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001606,-12.04,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001607,-11.83,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001608,-11.71,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001609,-11.71,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001610,-11.71,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001611,-11.71,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001612,-11.93,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001613,-11.98,1.28335,0
>>>> >>>>
>>>> >>>> 100112,001614,-11.04,1.28335,7
>>>> >>>>
>>>> >>>> the price keeped changing until 001602, then was stuck - as far as
>>>> I
>>>> >>>> could see in the file all the way until I restarted.
>>>> >>>>
>>>> >>>>
>>>> >>>> The following is an excerpt from the Eventreport file - it
>>>> contains the
>>>> >>>> end before the restart (the were no further events recorded)
>>>> >>>>
>>>> >>>> 09/30/12 23:59:05.196 IB API 2103: Market data farm connection is
>>>> >>>> broken:usfuture
>>>> >>>>
>>>> >>>> 09/30/12 23:59:21.450 IB API 2104: Market data farm connection is
>>>> >>>> OK:usfuture
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET.
>>>> >>>> Please empty deep book contents before applying any new entries.
>>>> (for id 3)
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 3 has
>>>> been
>>>> >>>> reset.
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET.
>>>> >>>> Please empty deep book contents before applying any new entries.
>>>> (for id 2)
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 2 has
>>>> been
>>>> >>>> reset.
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 IB API 317: Market depth data has been RESET.
>>>> >>>> Please empty deep book contents before applying any new entries.
>>>> (for id 1)
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 My JBookTrader Market data for book 1 has
>>>> been
>>>> >>>> reset.
>>>> >>>>
>>>> >>>> 10/01/12 00:14:04.896 IB API 2103: Market data farm connection is
>>>> >>>> broken:usfuture
>>>> >>>>
>>>> >>>> 10/01/12 00:14:42.166 IB API 2104: Market data farm connection is
>>>> >>>> OK:usfuture
>>>> >>>>
>>>> >>>> 10/01/12 00:15:48.860 IB API 1100: Connectivity between IB and
>>>> Trader
>>>> >>>> Workstation has been lost.
>>>> >>>>
>>>> >>>> 10/01/12 00:16:16.630 IB API 1102: Connectivity between IB and
>>>> Trader
>>>> >>>> Workstation has been restored - data maintained.
>>>> >>>>
>>>> >>>>
>>>> >>>> So, it looks like connectivity was lost and restored, but from this
>>>> >>>> point forward no price updates were recorded..
>>>> >>>>
>>>> >>>> Any idea what is going on, would be helpful.
>>>> >>>>
>>>> >>>> Cheers
>>>> >>>> Klaus
>>>> >>>>
>>>> >>>>
>>>> >>> --
>>>> >>> You received this message because you are subscribed to the Google
>>>> Groups
>>>> >>> "JBookTrader" group.
>>>> >>> To view this discussion on the web visit
>>>> >>> https://groups.google.com/d/msg/jbooktrader/-/O4d4rFkJ3F0J.
>>>> >>>
>>>> >>> To post to this group, send email to [email protected].
>>>> >>> To unsubscribe from this group, send email to
>>>> >>> [email protected].
>>>> >>>
>>>> >>> For more options, visit this group at
>>>> >>> http://groups.google.com/group/jbooktrader?hl=en.
>>>> >>
>>>> >>
>>>> >>
>>>> > --
>>>> > You received this message because you are subscribed to the Google
>>>> Groups
>>>> > "JBookTrader" group.
>>>> > To view this discussion on the web visit
>>>> > https://groups.google.com/d/msg/jbooktrader/-/tEYk3_95L5EJ.
>>>> >
>>>> > To post to this group, send email to [email protected].
>>>> > To unsubscribe from this group, send email to
>>>> > [email protected].
>>>> > For more options, visit this group at
>>>> > http://groups.google.com/group/jbooktrader?hl=en.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "JBookTrader" group.
>>>> To post to this group, send email to [email protected].
>>>> To unsubscribe from this group, send email to
>>>> [email protected].
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/jbooktrader?hl=en.
>>>>
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"JBookTrader" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/jbooktrader/-/VQInXSoseSUJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/jbooktrader?hl=en.
