[OSM-dev] osmosis stuck, possible to debug?

2011-06-22 Thread Stephan Knauss 
Hi, 

this afternoon my update process got stuck. Running on linux. according to 
ps it's in state Sl. 


It's stated with these arguments:
osmosis -q --rri --bc --simc --bc --write-xml-change - | 

I guess killing it is all it takes to recover. But is there any tracing I 
could enable now to detect why it got stuck and make it possible to fix the 
issue?
Attach a debugger to get a stacktrace or similar? 


Stephan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmosis stuck, possible to debug?

2011-06-22 Thread Frederik Ramm 

Hi,

On 06/22/11 18:51, Stephan Knauss wrote:

this afternoon my update process got stuck. Running on linux. according
to ps it's in state Sl.
It's stated with these arguments:
osmosis -q --rri --bc --simc --bc --write-xml-change - |
I guess killing it is all it takes to recover. But is there any tracing
I could enable now to detect why it got stuck and make it possible to
fix the issue?


I don't know, but when this happens it tends to happen to almost 
everybody. It happened to four machines where I run minutely updates, 
too. It's usually once every few months. I have no idea why.


Bye
Frederik



___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmosis stuck, possible to debug?

2011-06-22 Thread Stephan Knauss 

On 22.06.2011 19:12, Frederik Ramm wrote:

On 06/22/11 18:51, Stephan Knauss wrote:

this afternoon my update process got stuck.

I don't know, but when this happens it tends to happen to almost
everybody. It happened to four machines where I run minutely updates,
too. It's usually once every few months. I have no idea why.


I'm not used to all the debugging tools available. I was able to create 
the following stack trace using jstack


For mee it looks like it got stuck inside some method checking the 
server state. How is it supposed to handle network timeouts and such things?


Deadlock Detection:

No deadlocks found.

Thread 23590: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) 
@bci=14, line=158 (Interpreted frame)
 - 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
@bci=42, line=1987 (Interpreted frame)
 - org.openstreetmap.osmosis.core.store.DataPostbox.waitForUpdate() 
@bci=4, line=99 (Interpreted frame)
 - 
org.openstreetmap.osmosis.core.store.DataPostbox.consumeCentralQueue() 
@bci=33, line=157 (Interpreted frame)
 - org.openstreetmap.osmosis.core.store.DataPostbox.hasNext() @bci=15, 
line=235 (Interpreted frame)
 - org.openstreetmap.osmosis.core.buffer.v0_6.ChangeBuffer.run() 
@bci=6, line=73 (Interpreted frame)

 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 23589: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) 
@bci=14, line=158 (Interpreted frame)
 - 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
@bci=42, line=1987 (Interpreted frame)
 - org.openstreetmap.osmosis.core.store.DataPostbox.waitForUpdate() 
@bci=4, line=99 (Interpreted frame)
 - 
org.openstreetmap.osmosis.core.store.DataPostbox.consumeCentralQueue() 
@bci=33, line=157 (Interpreted frame)
 - org.openstreetmap.osmosis.core.store.DataPostbox.hasNext() @bci=15, 
line=235 (Interpreted frame)
 - org.openstreetmap.osmosis.core.buffer.v0_6.ChangeBuffer.run() 
@bci=6, line=73 (Interpreted frame)

 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 23588: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, 
byte[], int, int, int) @bci=0 (Interpreted frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 
(Interpreted frame)
 - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted 
frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, 
line=258 (Interpreted frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317 
(Interpreted frame)
 - 
sun.net.www.http.HttpClient.parseHTTPHeader(sun.net.www.MessageHeader, 
sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) 
@bci=51, line=687 (Interpreted frame)
 - sun.net.www.http.HttpClient.parseHTTP(sun.net.www.MessageHeader, 
sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) 
@bci=30, line=632 (Interpreted frame)
 - sun.net.www.protocol.http.HttpURLConnection.getInputStream() 
@bci=294, line=1195 (Interpreted frame)

 - java.net.URL.openStream() @bci=4, line=1010 (Interpreted frame)
 - 
org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(java.net.URL, 
java.lang.String) @bci=31, line=87 (Interpreted frame)
 - 
org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(java.net.URL) 
@bci=4, line=45 (Interpreted frame)
 - 
org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl() 
@bci=37, line=263 (Interpreted frame)
 - 
org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run() 
@bci=26, line=345 (Interpreted frame)

 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 23580: (state = BLOCKED)


Thread 23509: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 
(Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted 
frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 
(Interpreted frame)



Thread 23508: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 
(Interpreted frame)



Thread 23360: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Thread.join(long) @bci=38, line=1186 (Interpreted frame)
 - java.lang.Thread.join() @bci=2, line=1239 (Interpreted frame)
 - 
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager.waitForCompletion() 
@bci=44, line=75 (Interpreted frame)
 - 
org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion() 
@bci=32, line=140 (Interpreted frame)
 -

Re: [OSM-dev] osmosis stuck, possible to debug?

2011-06-22 Thread Stephan Knauss 

On 22.06.2011 21:00, Stephan Knauss wrote:

For me it looks like it got stuck inside some method checking the
server state. How is it supposed to handle network timeouts and such
things?


is osmosis setting a timeout at all? Is there any global setting in 
effect? getConnectTimeout() could return a value. In case this is 0 it's 
the problem we all experience with it waiting infinitely.


I guess it's this code:
http://svn.openstreetmap.org/applications/utils/osmosis/trunk/replication/src/org/openstreetmap/osmosis/replication/common/ServerStateReader.java 
		


stateStream = stateUrl.openStream();


according to Javadoc it's a shorthand for
 openConnection().getInputStream()


I propose to replace this with

URLConnection connection = stateUrl.openConnection();
connection.setReadTimeout(15*60*1000); // timeout 15 minutes
connection.setConnectTimeout(15*60*1000); // timeout 15 minutes
stateStream = connection.getInputStream();


Could this be tested by someone more familiar with the code than I am?

Stephan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmosis stuck, possible to debug?

2011-06-22 Thread Jeremy Adams 
On Wed, Jun 22, 2011 at 12:51 PM, Stephan Knauss o...@stephans-server.de 
wrote:
 Hi,
 this afternoon my update process got stuck. Running on linux. according to
 ps it's in state Sl.
 It's stated with these arguments:
 osmosis -q --rri --bc --simc --bc --write-xml-change - |
 I guess killing it is all it takes to recover. But is there any tracing I
 could enable now to detect why it got stuck and make it possible to fix the
 issue?
 Attach a debugger to get a stacktrace or similar?
 Stephan

It appears to have just happened again.  At least my TRAPI server did anyway...

-Jeremy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osmosis stuck, possible to debug?

2011-06-22 Thread Florian Lohoff 
On Wed, Jun 22, 2011 at 07:12:49PM +0200, Frederik Ramm wrote:
 Hi,
 
 On 06/22/11 18:51, Stephan Knauss wrote:
 this afternoon my update process got stuck. Running on linux. according
 to ps it's in state Sl.
 It's stated with these arguments:
 osmosis -q --rri --bc --simc --bc --write-xml-change - |
 I guess killing it is all it takes to recover. But is there any tracing
 I could enable now to detect why it got stuck and make it possible to
 fix the issue?
 
 I don't know, but when this happens it tends to happen to almost
 everybody. It happened to four machines where I run minutely
 updates, too. It's usually once every few months. I have no idea
 why.

I used to start osmosis with timeout a little unix tool which will kill
the called process once it takes to long.

Just put it to any value which always works so the update has at least
a chance to recover.

Flo
-- 
Florian Lohoff f...@zz.de
„Für eine ausgewogene Energiepolitik über das Jahr 2020 hinaus ist die
Nutzung von Atomenergie eine Brückentechnologie und unverzichtbar. Ein
Ausstieg in zehn Jahren, wie noch unter der rot-grünen Regierung
beschlossen, kommt für die nationale Energieversorgung zu abrupt.“
Angela Merkel CDU 30.8.2009


signature.asc
Description: Digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev