My answers in the text,
Hi Michaël,
My test file can now be successfully saved and read to/from shape and JML.
I made also a test with GDAL-dev versions with the created shapefile
ogrinfo -so -al datatype_test.shp
INFO: Open of `datatype_test.shp'
using driver `ESRI Shapefile' successful.
Layer name: datatype_test
Geometry: Point
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer64 (11.0)
tinyint_at: Integer64 (11.0)
integer_at: Integer64 (11.0)
long_attri: Real (21.0)
bigint_att: Real (21.0)
This one is a bit strange. Maybe they don't want to interpret N(21,0) as
Integer64
because not all N(21,0) are representable with an Integer64.
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
Notice that all the short integers are interpreted as long integers
(Integer64) and the long ones as Reals. Perhaps you should consider
to make the numbers a little bit shorter? According to this ticket
http://trac.osgeo.org/gdal/ticket/3615 ever a number with 10 digits
(10.0) can be too big as an Integer. I suppose that the biggest
Integer is 4,294,967,29. And numbers with 20 digits can be bigger
than Long integers if they are >18,446,744,073,709,551,615.
This is a difficult problem. At the moment, mapping from OJ data types
to shapefile data types do not need any knowledge about data itself
except for strings (to set string field length).
We can implement something similar to string type to set integer field
length to 10 or 11 only if it is needed, and keep length = 9 for
integers between -99 999 999 and 999 999 999
I guess that storing integers as N(11,0) is the reason why gdal
interpret them as Integer64 and not Integer32.
GDAL shp driver
http://www.gdal.org/drv_shapefile.html is behaving this way when it
creates numeric fields :
*
Integer fields without an explicit width are treated as width 9,
and extended to 10 or 11 if needed.
*
Integer64 fields without an explicit width are treated as width
18, and extended to 19 or 20 if needed.
I made some tests about what GDAL does in reading. It appears to
reports numbers only up to (9.0) as Integers and up to (18.0) as Long
Integers.
I wonder if it would be better to change the shapefile writer of
OpenJUMP to create Integer fields by default as (9.0) and change
format into (10.0) only if integer field contains values between
1,000,000,00 and 4,294,967,29. Bigger values than the upper limit
should not be accepted into integer field because they are invalid
everywhere.
OK, I'll have a look. Note that every solution has its drawback :
If an OJ layer contains Integers up to 100 and another layer contains
Integers greater than 1 000 000 000, they will be stored respectively in
a N(9,0) and N(10,0), then read back into OJ as Integer and Long. May
not be critical though.
There seems to be another issue with Long integers in the shapefiles.
Long integers can need up to 20 digits but the standard dBase format
has an 18 digit limit for numbers
http://www.clicketyclick.dk/databases/xbase/format/data_types.html.
<http://www.clicketyclick.dk/databases/xbase/format/data_types.html> Some
version has extended that to 20 numbers. Because of best possible
interoperability I think that OpenJUMP should create the Long fields
as (18.0) by default and (19.0) or (20.0) only if needed.
OK, this is a good argument.
I'll stick to 18 (and eventually test if more is needed).
Michaël
-Jukka Rahkonen-
------------------------------------------------------------------------
Michaël Michaud wrote:
> Hi Jukka,
> Thank you for the test and sorry for the exceptions.
I just completed with BIGINT, TIME and NUMERIC.
Shapefile driver will not really handle all these types. I've just handled
long and boolean in a specific way. Other types are just mapped to old
types.
> This is how new types are supposed to be converted to dbf then back
to OpenJUMP :
CHAR, VARCHAR, LONGVARCHAR, TEXT, STRING, OBJECT -> C -> STRING
FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL, BIGDECIMAL -> N(33,16) -> DOUBLE
TINYINT, SMALLINT, INTEGER -> N(11) -> INTEGER
LONG, BIGINT -> N(21) -> LONG
DATE, TIME, TIMESTAMP -> D -> DATE
BOOLEAN, BIT -> L -> BOOLEAN
The only data types that I've sometimes missed are boolean and long.
That's why I tried to map them in a way that can preserve type
information when
you save to dbf and back.
For other data types, my main concern is just to make the drivers
compatible with
the UI.
Michaël
Le 15/03/2015 18:26, Rahkonen Jukka (MML) a écrit :
Hi,
I made a test file with one point and one attribute of each
selectable data type. However, OpenJUMP is not totally ready for
handling all those.
Saving the JML file as shapefile stops to the following error:
java.lang.Exception: ShapefileWriter: unsupported AttributeType found
in featurecollection. : BIGINT
at
com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:537)
at
com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:292)
at
com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:73)
at
com.vividsolutions.jump.workbench.datasource.AbstractSaveDatasetAsPlugIn.run(AbstractSaveDatasetAsPlugIn.java:28)
at
com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:152)
at java.lang.Thread.run(Unknown Source)
Next I tried to edit the schema and remove the BIGINT attribute but
it was not so easy. Changes in the schema can be confirmed only after
removing all the attributes of the following data types first:
CHAR
VARCHAR
LONGVARCHAR
TEXT
BOOLEAN
BIT
SMALLINT
TINYINT
LONG
BIGINT
DECIMAL
NUMERIC
BIGDECIMAL
FLOAT
REAL
TIME
TIMESTAMP
The error if any of the above data types appears in the schema is like:
com.vividsolutions.jts.util.AssertionFailedException: Should never
reach here: VARCHAR
at
com.vividsolutions.jts.util.Assert.shouldNeverReachHere(Assert.java:122)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:557)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:286)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.applyChanges(ViewSchemaPlugIn.java:164)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.access$300(ViewSchemaPlugIn.java:76)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn$EditSchemaFrame$3.actionPerformed(ViewSchemaPlugIn.java:695)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.fireActionPerformed(SchemaPanel.java:686)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.applyButton_actionPerformed(SchemaPanel.java:676)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel$14.actionPerformed(SchemaPanel.java:447)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$200(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown
Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown
Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Here is how GDAL understands our new attribute types:
ogrinfo datatype_test.jml -al -so
Had to open data source read-only.
INFO: Open of `datatype_test.jml'
using driver `JML' successful.
Layer name: dttest
Geometry: Unknown (any)
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_attribute: String (0.0)
char_attribute: String (0.0)
varchar_attribute: String (0.0)
longvarchar_attribute: String (0.0)
text_attribute: String (0.0)
boolean_attribute: String (0.0)
bit_attribute: String (0.0)
smallint_attribute: String (0.0)
tinyint_attribute: String (0.0)
integer_attribute: Integer (0.0)
long_attribute: String (0.0)
bigint_attribute: String (0.0)
decimal_attribute: String (0.0)
numeric_attribute: String (0.0)
bigdecimal_attribute: String (0.0)
float_attribute: String (0.0)
double_attribute: Real (0.0)
real_attribute: String (0.0)
date_attribute: DateTime (0.0)
time_attribute: String (0.0)
timestamp_attribute: String (0.0)
object_attribute: String (0.0)
Most new types are converted into strings by now. I think I could
sponsor the GDAL development if someone writes the mapping between
OpenJUMP datatypes and GDAL datatypes. I am not sure what the GDAL
datatypes are but the following list might be right:
1. boolean (GDAL >= 2.0)
2. character(field_length). By default, field_length=1.
3. float(field_length)
4. numeric(field_length, field_precision)
5. smallint(field_length) : 16 bit signed integer (GDAL >= 2.0)
6. integer(field_length)
7. bigint(field_length), 64 bit integer, extension to SQL92 (GDAL >=
2.0)
8. date(field_length)
9. time(field_length)
10. timestamp(field_length)
Obviously 'string', and 'real' are supported and then I believe that
there is support for these three:
'integer list', 'double list' and 'string list'.
However, I have not yet realized if we need all these data types and
how generally they are supported in the surrounding GIS and computing
world.
-Jukka Rahkonen-
------------------------------------------------------------------------
Michaël Michaud wrote:
Hi,
I managed new attribute types introduced by Ede in shapefile and jml
drivers (BOOLEAN and LONG are managed as "Logical" and
"Numeric,21,0", and other types are managed as one of the former types).
Please use and test these new types, especially with shapefiles.
Next step is to introduce these types in PostGIS database driver.
Michaël
-------- Message transféré --------
Sujet : [jump-pilot:code] [r4337] - michaudm: Manage new attribute
types BOOLEAN and LONG in jml and shp drivers (also manage other
datatypes so that they can be written as one of the well-known
datatype instead of throwing an error message)
Date : Sat, 14 Mar 2015 16:55:29 +0000
De : Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net>
Répondre à : Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net>
Pour : Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net>
Manage new attribute types BOOLEAN and LONG in jml and shp drivers
(also manage other datatypes so that they can be written as one of
the well-known datatype instead of throwing an error message)
http://sourceforge.net/p/jump-pilot/code/4337/
<http://sourceforge.net/p/jump-pilot/code/4337>
------------------------------------------------------------------------
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/jump-pilot/code/
<https://sourceforge.net/p/jump-pilot/code>
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/
<https://sourceforge.net/auth/subscriptions>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
