[osmosis-dev] [PATCH] Remove changing=true from fixed-revision Ivy dependencies
In ivy.xml, the setting 'changing=true' tells Ivy that the
dependency's jar might change despite its version not changing. This
causes Ivy to always query the repository, even if the jar is
available in the local cache. This setting is often used to express a
dependency on the latest snapshot (e.g., nightly build) of a 3rd party
project.
Unfortunately, this setting will cause offline builds to fail if the
dependency comes from a non-local repository, even if the dependency
is available in the local cache. Thus, it should be avoided for
fixed-revision dependencies.
The versions of the osmpbf and netty dependencies do not represent
snapshots that could change over time, so this change removes
'changing=true from the dependency declarations.
---
pbf/ivy.xml |2 +-
replication-http/ivy.xml |2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/pbf/ivy.xml b/pbf/ivy.xml
index 6f6a0af..cebf4bf 100644
--- a/pbf/ivy.xml
+++ b/pbf/ivy.xml
@@ -27,7 +27,7 @@
dependency org=org.openstreetmap.osmosis name=osmosis-core rev=${project.version}
conf=compile-default changing=true/
dependency org=com.google.protobuf name=protobuf-java
rev=${dependency.version.protobuf} conf=compile-default/
- dependency org=crosby name=osmpbf rev=${dependency.version.osmpbf}
conf=compile-default changing=true/
+ dependency org=crosby name=osmpbf rev=${dependency.version.osmpbf}
conf=compile-default/
dependency org=org.openstreetmap.osmosis name=osmosis-xml rev=${project.version}
conf=test-default changing=true/
dependency org=org.openstreetmap.osmosis name=osmosis-testutil rev=${project.version}
conf=test-default changing=true/
diff --git a/replication-http/ivy.xml b/replication-http/ivy.xml
index 885275a..8844c6d 100644
--- a/replication-http/ivy.xml
+++ b/replication-http/ivy.xml
@@ -28,7 +28,7 @@
dependency org=org.openstreetmap.osmosis name=osmosis-replication rev=${project.version}
conf=compile-default changing=true/
dependency org=org.openstreetmap.osmosis name=osmosis-xml rev=${project.version}
conf=compile-default changing=true/
- dependency org=org.jboss.netty name=netty rev=${dependency.version.netty}
conf=compile-default changing=true/
+ dependency org=org.jboss.netty name=netty rev=${dependency.version.netty}
conf=compile-default/
dependency org=org.openstreetmap.osmosis name=osmosis-testutil rev=${project.version}
conf=test-default changing=true/
dependency org=junit name=junit rev=${dependency.version.junit}
conf=test-default/
--
1.7.9.2
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] calling DataPostbox.release() multiple times
On 2012-01-15 07:34, Brett Henderson wrote:
You're correct in saying that Releasable classes can be released
multiple times, however that is not typically how they're used in the
Osmosis pipeline.
Does it make sense to change the Releasable API definition to say that
release can only be called once? Or might that break multiple-sink
stages like merge?
DataPostbox is a bit tricky because it is the main synchronisation point
between multiple threads. Two threads communicating via a DataPostbox
perform handshakes during the initialize, complete, and release methods
(the process methods run mostly independently). It is the release
method where each threads tells the other thread that they are about to
exit. DataPostbox does technically support multiple calls to release,
but it would require each thread to restart after the release calls and
this isn't supported (or even desirable) by the Osmosis pipeline.
...
What changes would you suggest? I believe it's working as I originally
intended, but that's not to say there isn't a better way :-)
Looking at the code a bit more I think it might only need minor changes.
I had thought that release() and outputRelease() changed the state to
equal that of a freshly constructed DataPostbox (to support reuse?), but
that's not true: inputExit and outputExit are false after construction
but true after release(). So simply checking inputExit at the beginning
of release() should take care of it. See the attached (untested) patch.
An alternative is to increase the burden on the users of DataPostbox.
They could be required to either ensure that the output thread restarts
or avoid calling DataPostbox.release() once the output thread exits.
For example, EntityBuffer.run() could set an 'outputThreadIsRunning'
boolean at the top and clear it at the end. Then EntityBuffer.release()
would be modified to only call buffer.release() if outputThreadIsRunning
is true.
-Richard
From 86c93d88e011ab76b79737c6185bd138f2eced76 Mon Sep 17 00:00:00 2001
From: Richard Hansen rhan...@bbn.com
Date: Sun, 15 Jan 2012 15:13:01 -0500
Subject: [PATCH] Do nothing in DataPostbox.release() if already released
The Releasable interface specification says that release() may be
called multiple times. Before, if DataPostbox.release() was called a
second time, it would block forever waiting for the output thread
(which would no longer exist) to release. Now subsequent calls to
release() are a no-op, preventing the lockup.
---
.../osmosis/core/store/DataPostbox.java|5 +
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java b/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java
index 372de7f..415ee40 100644
--- a/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java
+++ b/core/src/main/java/org/openstreetmap/osmosis/core/store/DataPostbox.java
@@ -335,6 +335,11 @@ public class DataPostboxT implements Initializable {
lock.lock();
try {
+ // support release() being called multiple
+ // times (do nothing if already released)
+ if (inputExit)
+return;
+
// If release is being called without having completed successfully,
// it is an error condition.
if (!inputComplete) {
--
1.7.8.2
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] [PATCH] --log-progress --buffer results in OsmosisRuntimeException: initialize has not been called
Hi all,
With the latest Git revision (43089ed), I get an OsmosisRuntimeException
when I use --log-progress followed by --buffer. For example:
osmosis -v \
--rb massachusetts.osm.pbf --sort \
--rb connecticut.osm.pbf --sort \
--merge --log-progress --buffer --wb _combined.osm.pbf
...
org.openstreetmap.osmosis.core.OsmosisRuntimeException: initialize has
not been called
at
org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:288)
at
org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.process(EntityBuffer.java:48)
at
org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.process(EntityProgressLogger.java:70)
at
org.openstreetmap.osmosis.core.sort.v0_6.EntitySorter.complete(EntitySorter.java:71)
at
crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)
at
crosby.binary.file.BlockInputStream.process(BlockInputStream.java:37)
at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
at java.lang.Thread.run(Thread.java:679)
If I remove the --log-progress argument, it works as expected.
I chased this down to a missing call to sink.initialize() in
EntityProgressLogger.initialize(). Attached is a patch that I believe
fixes this bug.
-Richard
From 33cabe82f3a55dba2f3f5942cbe021177a8db493 Mon Sep 17 00:00:00 2001
From: Richard Hansen rhan...@bbn.com
Date: Sat, 14 Jan 2012 15:51:33 -0500
Subject: [PATCH] Add missing call to sink.initialize() in
EntityProgressLogger
Without this change, when --log-progress is followed by --buffer, an
OsmosisRuntimeException is thrown by DataPostbox.put() complaining
that DataPostbox.initialize() has not been called:
osmosis -v \
--rb massachusetts.osm.pbf --sort \
--rb connecticut.osm.pbf --sort \
--merge --log-progress --buffer --wb _combined.osm.pbf
...
org.openstreetmap.osmosis.core.OsmosisRuntimeException: initialize has not been called
at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:288)
at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.process(EntityBuffer.java:48)
at org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.process(EntityProgressLogger.java:70)
at org.openstreetmap.osmosis.core.sort.v0_6.EntitySorter.complete(EntitySorter.java:71)
at crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)
at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:37)
at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
at java.lang.Thread.run(Thread.java:679)
---
.../core/progress/v0_6/EntityProgressLogger.java |1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/core/src/main/java/org/openstreetmap/osmosis/core/progress/v0_6/EntityProgressLogger.java b/core/src/main/java/org/openstreetmap/osmosis/core/progress/v0_6/EntityProgressLogger.java
index a6c4de1..b531e14 100644
--- a/core/src/main/java/org/openstreetmap/osmosis/core/progress/v0_6/EntityProgressLogger.java
+++ b/core/src/main/java/org/openstreetmap/osmosis/core/progress/v0_6/EntityProgressLogger.java
@@ -49,6 +49,7 @@ public class EntityProgressLogger implements SinkSource {
*/
public void initialize(MapString, Object metaData) {
progressTracker.initialize();
+ sink.initialize(metaData);
}
--
1.7.8.2
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] calling DataPostbox.release() multiple times
Hi all, I believe I have found a bug in DataPostbox.release(). (I'm using git master, currently 43089ed.) According to the javadoc comments, Releaseable.release() can be called multiple times. However, DataPostbox.release() can only be called once. If it is called a second time, it blocks forever waiting for the output thread (which no longer exists) to release. I noticed this because the pbf reader can call release() twice, triggering the lockup. To recreate, you can run the following: osmosis --read-pbf /dev/null --buffer --write-null The two calls to DataPostbox.release() have the following stack traces: java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1266) at org.openstreetmap.osmosis.core.store.DataPostbox.release(DataPostbox.java:335) at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.release(EntityBuffer.java:64) at crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:36) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:37) at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45) at java.lang.Thread.run(Thread.java:679) java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1266) at org.openstreetmap.osmosis.core.store.DataPostbox.release(DataPostbox.java:335) at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.release(EntityBuffer.java:64) at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:50) at java.lang.Thread.run(Thread.java:679) Deleting line 36 of pbf/src/main/java/crosby/binary/osmosis/OsmosisBinaryParser.java will cause DataPostbox.release() to be called only once, but that just avoids the bug. I believe a proper fix to DataPostbox.release() will require a minor redesign of the DataPostbox lifetime management code. I'm not familiar enough with the code to feel comfortable making such a change. I can take a crack at it, but I'm hoping an expert has the time to take a look. Thanks, Richard ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
