Hi all,
yesterday we chatted a bit about how the suite
installation process can include a step that allows
mass import of all the shapefiles contained in a directory.
I've attached the full logs if someone wants to
re-read them.
General importer spec
--------------------------------------------------
The user somehow specifies a directory containing
shapefiles, and the process is supposed to configure
them all in the GeoServer catalog automatically
setting:
1) the SRS
2) a default style
3) the bounding box
The bounding box is read from the shapefile header,
the default style is chosen from the geometry type
(should it be a copy of the default, or the
default itself? I.e., should each layer end up
with its own personal style or all polygon ones
share the same polygon.sld style?)
The SRS is tricky in that we need to come up
from an official EPSG code, the shapefiles might
come without a .prj from which we cannot
extract a EPSG code, or don't have a .prj at
all.
Options here:
a) the user specifies a default EPSG code for
the cases in which we cannot determine a
native EPSG code
b) the import process fails if no .prj is found,
otherwise if it cannot determine an EPSG code
it will just set 4326 and force a "reproject
native to declared" srs handling
Option a) is nicer to the user (I have seen a whole
lot of shapefiles without .prj around) but requires
some decent UI to choose the default EPSG code
(something that allows to drill into the 3000+ available
codes)
Regardelss of the details, the moving parts needed
to pull this off are a tool to write the GeoServer
configruation plus the shapefile datastores (or better,
the directory datastore) so that we can gather the
shapefile structural information (we need attributes
too).
GeoServer itself has all of the above, anything
else will have to reinvent the wheel.
So it is suggested that whatever the UI is,
a running GeoServer ends up doing the heavy lifting.
This means we need to ensure GeoServer is running
once we start this import process.
Importer UI
------------------------------------------------
The importer UI should run during/right after
the installer did its job. It should also be
possible to run it multiple times to import
more than one set of shapefiles.
The latter should receive some more consideration imho,
my experience is that people tend to lump together
files by project. If they want to import multiple
projects we should consider allowing people to
keep them separate in GS as well (think organisation
issues and name conflicts).
A way to do so could be to ask them to provide
the URL of their site and a "project" name, and build
a workspace definition out of that.
For example, "www.google.com", "foo" would become
the "foo" workspace whose namespace is
"http://www.google.com/foo"; and the shapefile "xyz"
would become the "foo:xyz" layer.
Besides this, the UI should provide a directory
chooser and an eventual EPSG code chooser (depends
on which of the SRS handling options we choose).
Clarification: the UI needs to choose a directory
being on the same machine GeoServer is running.
Options considered:
- an NSIS panel asking just for the directory,
which ends up making a REST call to GeoServer
passing the path, GeoServer does the rest.
Pros:
* native, integrated in the installer
Cons:
* can't be rerun after the install
- a Swing app asking the directory and eventually
providing a simple SRS chooser, which ends
up making a REST call to GeoServer passing
the path and the SRS
Pros:
* can be rerun
* we have a choice of widgets for both the tasks
Cons:
* a bit on the heavy side
- a Site Specific Browser running a pure HTML/JS
that does ask for the local directory and the
SRS
Pros:
* can be rerun
* can be made to look very stylish
Cons:
* requires a js coder and some access to the
local filesystem
* we need a SSB solution that allows local FS
access
- a Site Specific Browser running a Wicket wizard
that is actually part of GeoServer itself
Pros:
* can be rerun, even from different hosts
* can be integrated as part of the normal
GeoServer experience
Cons:
* the directory chooser control is not part
of the Wicket widgets, I made one in my
spare time but it does not look good,
so some design love is needed
If we go the SSB road, mind we need to include
a SSB in the installer, and we'll need a different
one for each platform (WebRunner works multiplatform,
but mind, it adds 8MB to the download).
I was wondering if we can't just open a plain browser
window (less stylish for sure).
Time wise, the quickest for me to develop is the
Swing one, I can make the Swing UI in probably around
half a day. The plain NSIS + rest call is quick too,
not sure we provide good feedback out of that
(a summary of how the import process did would be nice).
The one I like the best is the plain browser window
with the Wicket UI running in it.
Cheers
Andrea
--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.
(18:27:13) bmmpxf [n=m...@nyc01.limewire.com] è entrato nel canale.
(18:27:19) ***rpenate bristles with ChanServ power!
(18:27:29) aaime: lol
(18:27:33) bmmpxf: "By the power of Freenode!"
(18:27:38) jdeolive [n=jdeol...@s0106000f660192a4.cn.shawcable.net] è entrato
nel canale.
(18:27:40) sophiap: I can feel the static electricity
(18:27:58) cholmes [n=chol...@nyc01.limewire.com] è entrato nel canale.
(18:28:16) aaime: so, rpenate, can you explain how is the installer option
better?
(18:28:27) bmmpxf: define "installer option"
(18:28:29) bmmpxf: please?
(18:28:36) aaime: as in, import during the install process
(18:28:40) aaime: as opposed in the gs ui
(18:29:02) bmmpxf: rpenate, you have the floor
(18:29:33) rpenate: well
(18:29:58) rpenate: if our goal is to nail a "first-time experience" then the
installer provides a clear, linear, and concise way of getting from point A to
point B
(18:30:09) rpenate: much like the setup process at that start of Mac OS X
(18:30:30) rpenate: asking the user to deviate into the GS admin interface is a
detour into the back alley behinds point C and D
(18:30:42) rpenate: that breaks the flow and leaves room for the user to get
lost
(18:31:12) rpenate: (the OSX example is the first boot after install)
(18:31:13) aaime: rpenate, what if gs starts and the ui pops up so that you are
right in front of the gs ui at the end of the install process?
(18:31:34) aaime: I was thinking these wizards could be part of the home page
actions
(18:31:59) rpenate: don't get me wrong, i think this would be great in GS,
especially as a Welcome page action
(18:32:02) aaime: the thing is, I need gs itself to make a configuration anyways
(18:32:15) sophiap: eclipse handles this by providing a screen that provides
options: take a tour, work through a tutorial, or get busy
(18:32:38) iwillig [n=iwil...@nyc01.limewire.com] è entrato nel canale.
(18:32:40) aaime: or else I'll have to roll a separate config app that has a
Swing UI and that is a half GS running outside of a container (not sure it
would work either)
(18:32:44) sophiap: exactly, a welcome page with options
(18:32:49) rpenate: bmmpxf: would it make sense to have it be a dumping point
from the installer? or one of the options in the installer (along side,
Install, Repair, Uninstall, Configure, etc?)
(18:32:57) bmmpxf: My my perspective, I would like the import wizard to be
spawned immediately upon successful installation, but I also would like users
to be able to get back to that stage at a later time should they decide to
import more shapefiles later.
(18:33:04) bmmpxf: From ym perspective...
(18:33:08) cholmes: what about a single site browser that just has the import
wizard pages?
(18:33:23) aaime: cholmes, I can do that as well
(18:33:33) aaime: the trick is recognizing when GS is up and running
(18:33:37) bmmpxf: rpenate: I would like to have multiple options, yes, not
sure how possible that is, but would make the most sense.
(18:33:41) rpenate: I actually think considering the Welcome page as a part of
the install process is a smart idea
(18:33:46) cholmes: we start geoserver during install process
(18:33:54) aaime: we don't right now
(18:34:01) bmmpxf: that's easy
(18:34:05) bmmpxf: to start HS after install
(18:34:06) cholmes: but we're making a new install process
(18:34:07) bmmpxf: er GS
(18:34:24) cholmes: for OpenGeo Suite
(18:34:30) aaime: the harder part is to recognize if/when GS is ready to serve
requests
(18:34:37) bmmpxf: yes, perhaps this channel should be called #oginstaller :/
(18:34:43) aaime: I guess we can do a pure javascript page that polls it somehow
(18:35:47) sophiap: I've always thought that W*S servers should have a
heartbeat request
(18:36:07) cholmes: how would that work sophiap?
(18:36:26) awright [n=awri...@nyc01.limewire.com] è entrato nel canale.
(18:36:27) aaime: cholmes, the idea was to ahve the browser look like a normal
window, that is, removing all the window decorations? Not sure we can do it in
a plain window, I know it's possible to do that for popup
(18:36:29) aaime: rpenate?
(18:36:44) aaime: (window decoration -> I mean url bar and whatnot)
(18:36:52) sophiap: I used to have a class that would send a getcaps request to
make sure the endpoints were live
(18:36:54) rpenate: you mean wrap it in an SSB?
(18:37:08) rpenate: I don't know how well that integrates with Mike's installer
stuff
(18:37:11) aaime: SSB == ?
(18:37:21) bmmpxf: SSB would be great actually.
(18:37:31) rpenate: http://en.wikipedia.org/wiki/Site-specific_browser
(18:37:34) sophiap: but I think that there should be a request that sends back
the status, but this is all an aside
(18:38:26) bmmpxf: The SSB could be a dashboard, including GS On/Off + a button
for the import wizard, which could spawn whatever we wanted (be in a browser,
or another NSIS script)
(18:38:35) rpenate: whit has been doing some work with them, specifically
Titanium
(18:38:41) cholmes: So yeah, the reason the path of using nsis scripts to write
out config directly was appealing to me was that it wouldnt' depend on
geoserver being up.
(18:39:07) cholmes: but doing things in wicket is appealing because it'd be
reusable, and _may_ be less work?
(18:39:10) bmmpxf: cholmes: But then we couldn't use ReST, and would have to
hack catalogs manuall, right?
(18:39:15) cholmes: yes
(18:39:16) aaime: cholmes, unfortunately the only supported way to write a gs
config is to have gs up
(18:39:31) aaime: our config format does not have a schema, and it's subject to
change
(18:39:38) bmmpxf: What about an NSIS script that requires GS to be up and uses
ReST to make catalog changes?
(18:39:38) cholmes: well, this would be an unsupported way
(18:39:43) cholmes: right, but we'd test it each time.
(18:39:53) aaime: bmmpxf, that would requires us to write a rest client, we
have none
(18:39:59) cholmes: if nsis requires gs to be up and running then it loses most
of its appeal to me
(18:39:59) bmmpxf: oh right
(18:40:06) bmmpxf: oh
(18:40:09) bmmpxf: just a though
(18:40:10) aaime: besides, it keeps on being more work from where I stand
(18:40:11) bmmpxf: t
(18:40:12) cholmes: might as well do ssb with gs wicket at that point
(18:40:22) sophiap: yep
(18:40:25) cholmes: which keeps on being more work aaime?
(18:40:37) aaime: anything done with an external app as opposed from inside gs
(18:40:53) cholmes: Note the goal of this discussion is to figure out how to do
batch import in the opengeo suite installer with the _least_ amount of work.
(18:40:58) aaime: (mostly because we dont' have a nice java rest client)
(18:42:18) cholmes: So the question seems to be 'use nsis to hack gs config'
vs. 'ssb on wicket import ui'
(18:42:20) aaime: well, with an external app I could avoid having to work onthe
file browser thing
(18:42:30) aaime: directry choosers for swing apps are commonplace
(18:42:36) cholmes: not doing ssb wouldn't meet the UI experience we're working
for
(18:42:46) aaime: (because, no matetr what, we'd have to make that external
thing in java, no?)
(18:43:12) cholmes: I think we may be able to accomplish it with nsis scripts?
(18:43:21) cholmes: What do you see as blocker preventing that?
(18:43:28) cholmes: Nsis can select a directory no problem
(18:43:49) aaime: we also need to input a default epsg code
(18:43:51) cholmes: and can script file writing/updating I believe
(18:44:03) aaime: we can do that with nsis, but say goodby to a chooser
(18:44:09) aaime: the user would have t know the epsg code by heart
(18:44:10) bmmpxf: chooser?
(18:44:14) bmmpxf: oh
(18:44:25) aaime: bmmpxf, how do we create a chooser for the epsg database in
nsis?
(18:44:42) cholmes: just use 4326
(18:44:45) aaime: usualy people drill down using state names
(18:44:45) cholmes: reproject
(18:44:47) cholmes: from native
(18:44:58) aaime: assuming there is a native tos tart with
(18:45:04) bmmpxf: Hmmm, not sure, I think there's a drop down list option.
(18:45:06) aaime: which does not happen very often
(18:45:09) cholmes: no native then batch won't work...
(18:45:21) bmmpxf: If there's no native projection, then we can have the shp
import error out
(18:45:24) bmmpxf: right
(18:45:49) aaime: can nsis make network calls?
(18:46:03) cholmes: if people want to configure that then they can actually go
to the gs UI and input it in the normal way
(18:46:05) bmmpxf: A bigger question is hwo we take a shapefile and match it to
a point/line/polygon style
(18:46:10) aaime: we could make a rest front end for the mass importer
(18:46:14) cholmes: why do you need network calls?
(18:46:25) aaime: rest config :)
(18:46:43) aaime: how do I generate the geoserver config files?
(18:46:51) aaime: I guess we'd have to do a rest config call
(18:46:53) cholmes: rest front end has no appeal to me, since the win to me of
doing a desktop thing is non-network calls.
(18:47:07) cholmes: you'd start with a gs data directory
(18:47:17) cholmes: and script in the stuff you need to add
(18:47:21) aaime: look, whatever does it needs to open the shapefiles with
geotoosl to figure out crs and geometry type
(18:47:39) aaime: there is no way nsis can do that
(18:48:14) aaime: if it's an external app, it needs to blindly create the xml
files and use the geotools libs to to all teh crs/geom type lookups
(18:48:22) iwillig ha abbandonato il canale (quit: "Leaving.").
(18:48:26) whit [n=w...@c-69-244-203-243.hsd1.tn.comcast.net] è entrato nel
canale.
(18:48:26) aaime: or to use the rest api to call into gs to make the configs
(18:48:28) cholmes: I wonder if that can somehow happen after gs starting up.
(18:49:13) aaime: what I was proposing above, as an alternative to the SSB, is
that nsis does a single network call to a rest api that does all the heavy
lifting
(18:49:18) aaime: by working _inside_ geoserver
(18:49:46) aaime: http://.../rest/massConfigure?path=...
(18:49:49) bmmpxf: aaime: I was leaning towards that. I know that NSIS can use
cURL, not sure if that's the hook we need.
(18:50:12) ***bmmpxf stresses that he is _not_ an NSIS expert, only moderately
experienced.
(18:50:25) aaime: we could if we redistribute curl along with the package, but
I have no idea of how the nsis part would look like
(18:50:41) ***aaime just tweaked a little the existing nsis scripts, never used
nsis before
(18:51:35) aaime: doing an SSB with the Wicket UI looks more user friendly and
more reusable from the interative user pov
(18:52:03) aaime: doing NSIS + a mass importer rest api takes less time, more
reusable for people programming stuff
(18:52:16) aaime: but the user will be a sitting duck if he needs to import a
second directory
(18:52:28) bmmpxf: rerun the importer? :)
(18:52:41) bmmpxf: Multiple directories select?
(18:52:43) aaime: if it's driven by NSIS installer, how do you do that?
(18:53:07) aaime: bmmpxf, how many times you realized you needed to work on one
more item after the wizard was done?
(18:53:15) bmmpxf: I was thinking the SSB would be a dashboard that would spawn
the importer...
(18:53:17) aaime: a installer is the kind of wizard people do not like to rerun
(18:53:59) bmmpxf: Remember, that there would be two NSIS scripts in my vague
imaginging of this. An installer and an importer. Both using NSIS, one
calling the other, but both able to run independently.
(18:54:09) bmmpxf: yes, I agree aaime
(18:54:40) aaime: ok, so doing it inside nsis is not a good optoin it seems
(18:55:11) ***bmmpxf really is fine however we want to accomplish this...
(18:56:24) aaime: I vote for a wicket UI based wizard, even if it's more work
(18:56:52) bmmpxf: how long do you think that would take
(18:56:53) bmmpxf: ?
(18:56:53) aaime: second choice, a serate Swing app that uses a directory
chooser and maybe a drop down wtih the default srs and that does a rest call
(18:57:08) aaime: to geoserver, which actually does the mass import
(18:57:33) aaime: making a decent look file browser in wicket will take a few
days I guess
(18:57:42) ***whit doesn't see why you would need to build any ui in java if
you were using a SSB... wouldn't it just be html and js?
(18:58:13) aaime: because it seems I am the one that needs to do it
(18:58:28) aaime: and if I need to do it in JS I first need to learn it
(18:58:36) whit: gotcha
(18:58:50) ***rpenate is slightly confused...
(18:58:51) aaime: (studied it once, run away screaming)
(18:59:00) bmmpxf: ha!
(18:59:02) aaime: anyways, how do you do a dreictory chooser in js?
(18:59:19) whit: upload dialogue that comes with html forms?
(18:59:26) aaime: (to broser the local file system to choose a directory0
(18:59:38) aaime: whit, it has to choose a directory, not a file
(18:59:50) ***rpenate notes that the people at 427 Broadway have a mandatory
building-wide meeting right now....
(18:59:56) whit: depending on the SSB, some give you access to the filesystem
(19:00:16) rpenate: ok, this is getting too abstract
(19:00:25) aaime: anyways, if someone wants to do it pure java way I can make
the rest call that does the actual work
(19:00:29) rpenate: we're only dealing with the Windows case
(19:00:43) ***whit apologizes and butts out ;)
(19:00:55) aaime: (sorry, I meant the pure javascript way)
(19:01:01) rpenate: what are the options we've settled on?
(19:01:21) aaime: so far on the table we have SSB + wicket UI
(19:01:28) aaime: or swing app +
(19:01:30) aaime: rest call
(19:01:35) aaime: or html + js
(19:01:39) aaime: + rest call
(19:01:54) aaime: I can work on 1 and 2, and half of 3
(19:02:01) rpenate: bmmpxf: how would those interact with NSIS?
(19:02:02) aaime: the quickest is 2
(19:02:30) bmmpxf: rpenate: I thought this was in lieu of NSIS?
(19:02:40) whit ha abbandonato il canale (quit: ).
(19:02:45) rpenate: ok, but how does it integrate with the full stack install?
(19:02:52) rpenate: excuse me, OGS install
(19:03:14) aaime: as a post install app that gets run when the installer is
done?
(19:03:20) bmmpxf: After the installer is done, on the final screen, a check
box for "Import shapefiles" or something, sould spawn...whatever it is that we
create
(19:03:41) rpenate: ok
(19:03:45) bmmpxf: That spawned app could be rerun in the future from a link on
the start menu
(19:04:18) rpenate: ok
(19:04:26) aaime: mind, option 1 has the distinct advantage of being able to
run on all OS and even from a remote host
(19:04:54) aaime: so it's usable even after the first bite of gs
(19:05:11) rpenate: well, i have no qualifications to decide among those options
(19:05:13) rpenate: :)
(19:05:30) bmmpxf: jdeolive: Do you have any thoughts to add?
(19:05:32) aaime: np, I'm just adding items for the decision
(19:06:11) jdeolive: bmmpxf: unfortunately i have not really been up on the
conversation... :(
(19:06:19) ***jdeolive is putting out too many fires at once
(19:06:27) jdeolive: i will try to review and perhaps voice my thoughts in email
(19:06:29) bmmpxf: no prob, just saw you were int eh channel and silent
(19:06:47) bmmpxf: maybe we (aaime) can craft an email from this conversation?
(19:06:59) bmmpxf: if aaime wants, that is
(19:07:16) aaime: I can... but we don't have a decision
(19:07:34) aaime: so I send the mail and we think about that for some time?
(19:07:39) bmmpxf: true, but we have fleshed out the options... <shrug>
(19:16:42) whit [n=w...@adsl-19-216-103.bna.bellsouth.net] è entrato nel
canale.
(19:25:33) rpenate: i take it this meeting is over?
(19:31:02) aaime: I assumed as much, yeah
(19:31:37) rpenate: k
(20:03:18) whit ha abbandonato il canale (quit: Read error: 104 (Connection
reset by peer)).
(20:04:41) whit [n=w...@adsl-19-216-103.bna.bellsouth.net] è entrato nel
canale.
(20:25:36) cholmes ha abbandonato il canale (quit: Read error: 110 (Connection
timed out)).
------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
