Jukka,
In last revision, OJ checks the dataset first
If there is no value larger than 999 999 999 or lesser than -99 999 999,
it should use N(9,0)
but if the integer attribute contains values needing more than 9 digits,
it will choose the
field width accordingly (10 or 11).
The same should happen for longs (18 by default and 19 or 20 or 21 if
needed).
I made a small test to confirm that a integer attribute containing a
simgle value '123456789'
is saved as N(9,0) and read back as an Integer.
If you have a counter example, please, send it to me in jml format.
Michaël
Le 17/03/2015 11:34, Rahkonen Jukka (MML) a écrit :
Hi,
This must be some programming language magic, but even I can see in
your code for r4341 and I on testing with corresponding snapshot:
else if (maxlength <= 9) fields[f] = new DbfFieldDef(columnName,
'N', 9, 0);
OJ is still creating Integer field as (10.0). As a result a roundtrip
with OpenJUMP is now changing data type write into shp -> open back
with OJ -> what used to be integer is now Long.
In the same way, Long is saved as (19.0) even the plan was to make it
(18.0). I have verified this with OpenOffice Calc which is nice
because I can also edit the format by hand, and with GDAL:
This is how GDAL sees the fields now:
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer (6.0)
tinyint_at: Integer (6.0)
integer_at: Integer64 (10.0)
long_attri: Real (19.0)
bigint_att: Real (19.0)
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
-Jukka-
**Michaël Michaud wrote:
Jukka,
OK, I tried to implement it more or less as described in your previous
mail,
One other drawback with this change is that all previous shapefiles
containing integers and saved as N(11,0) will now be read back as longs.
Let me now if someone think this can be a problem,
Michaël
Le 16/03/2015 10:39, Rahkonen Jukka (MML) a écrit :
Hi Michaël,
My test file can now be successfully saved and read to/from shape
and JML.
I made also a test with GDAL-dev versions with the created shapefile
ogrinfo -so -al datatype_test.shp
INFO: Open of `datatype_test.shp'
using driver `ESRI Shapefile' successful.
Layer name: datatype_test
Geometry: Point
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer64 (11.0)
tinyint_at: Integer64 (11.0)
integer_at: Integer64 (11.0)
long_attri: Real (21.0)
bigint_att: Real (21.0)
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
Notice that all the short integers are interpreted as long
integers (Integer64) and the long ones as Reals. Perhaps you
should consider to make the numbers a little bit shorter?
According to this ticket
http://trac.osgeo.org/gdal/ticket/3615ever a number with 10 digits
(10.0) can be too big as an Integer. I suppose that the biggest
Integer is 4,294,967,29. And numbers with 20 digits can be bigger
than Long integers if they are >18,446,744,073,709,551,615.
GDAL shp driver
http://www.gdal.org/drv_shapefile.html is behaving this way when
it creates numeric fields :
·Integer fields without an explicit width are treated as width 9,
and extended to 10 or 11 if needed.
·Integer64 fields without an explicit width are treated as width
18, and extended to 19 or 20 if needed.
I made some tests about what GDAL does in reading. It appears to
reports numbers only up to (9.0) as Integers and up to (18.0) as
Long Integers.
I wonder if it would be better to change the shapefile writer of
OpenJUMP to create Integer fields by default as (9.0) and change
format into (10.0) only if integer field contains values between
1,000,000,00 and 4,294,967,29. Bigger values than the upper limit
should not be accepted into integer field because they are invalid
everywhere.
There seems to be another issue with Long integers in the
shapefiles. Long integers can need up to 20 digits but the
standard dBase format has an 18 digit limit for numbers
http://www.clicketyclick.dk/databases/xbase/format/data_types.html.<http://www.clicketyclick.dk/databases/xbase/format/data_types.html>
Some
version has extended that to 20 numbers. Because of best possible
interoperability I think that OpenJUMP should create the Long
fields as (18.0) by default and (19.0) or (20.0) only if needed.
-Jukka Rahkonen-
------------------------------------------------------------------------
Michaël Michaud wrote:
> Hi Jukka,
> Thank you for the test and sorry for the exceptions.
I just completed with BIGINT, TIME and NUMERIC.
Shapefile driver will not really handle all these types. I've just
handled
long and boolean in a specific way. Other types are just mapped to
old
types.
> This is how new types are supposed to be converted to dbf then
back to OpenJUMP :
CHAR, VARCHAR, LONGVARCHAR, TEXT, STRING, OBJECT -> C ->
STRING
FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL, BIGDECIMAL -> N(33,16) ->
DOUBLE
TINYINT, SMALLINT, INTEGER -> N(11) -> INTEGER
LONG, BIGINT -> N(21) -> LONG
DATE, TIME, TIMESTAMP -> D -> DATE
BOOLEAN, BIT -> L -> BOOLEAN
The only data types that I've sometimes missed are boolean and long.
That's why I tried to map them in a way that can preserve type
information when
you save to dbf and back.
For other data types, my main concern is just to make the drivers
compatible with
the UI.
Michaël
Le 15/03/2015 18:26, Rahkonen Jukka (MML) a écrit :
Hi,
I made a test file with one point and one attribute of each
selectable data type. However, OpenJUMP is not totally ready
for handling all those.
Saving the JML file as shapefile stops to the following error:
java.lang.Exception: ShapefileWriter: unsupported
AttributeType found in featurecollection. : BIGINT
at
com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:537)
at
com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:292)
at
com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:73)
at
com.vividsolutions.jump.workbench.datasource.AbstractSaveDatasetAsPlugIn.run(AbstractSaveDatasetAsPlugIn.java:28)
at
com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:152)
at java.lang.Thread.run(Unknown Source)
Next I tried to edit the schema and remove the BIGINT
attribute but it was not so easy. Changes in the schema can be
confirmed only after removing all the attributes of the
following data types first:
CHAR
VARCHAR
LONGVARCHAR
TEXT
BOOLEAN
BIT
SMALLINT
TINYINT
LONG
BIGINT
DECIMAL
NUMERIC
BIGDECIMAL
FLOAT
REAL
TIME
TIMESTAMP
The error if any of the above data types appears in the schema
is like:
com.vividsolutions.jts.util.AssertionFailedException: Should
never reach here: VARCHAR
at
com.vividsolutions.jts.util.Assert.shouldNeverReachHere(Assert.java:122)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:557)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:286)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.applyChanges(ViewSchemaPlugIn.java:164)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.access$300(ViewSchemaPlugIn.java:76)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn$EditSchemaFrame$3.actionPerformed(ViewSchemaPlugIn.java:695)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.fireActionPerformed(SchemaPanel.java:686)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.applyButton_actionPerformed(SchemaPanel.java:676)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel$14.actionPerformed(SchemaPanel.java:447)
at javax.swing.AbstractButton.fireActionPerformed(Unknown
Source)
at
javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at
javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown
Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at
java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at
java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown
Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$200(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown
Source)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown
Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown
Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at
java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown
Source)
at
java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at
java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown
Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Here is how GDAL understands our new attribute types:
ogrinfo datatype_test.jml -al -so
Had to open data source read-only.
INFO: Open of `datatype_test.jml'
using driver `JML' successful.
Layer name: dttest
Geometry: Unknown (any)
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_attribute: String (0.0)
char_attribute: String (0.0)
varchar_attribute: String (0.0)
longvarchar_attribute: String (0.0)
text_attribute: String (0.0)
boolean_attribute: String (0.0)
bit_attribute: String (0.0)
smallint_attribute: String (0.0)
tinyint_attribute: String (0.0)
integer_attribute: Integer (0.0)
long_attribute: String (0.0)
bigint_attribute: String (0.0)
decimal_attribute: String (0.0)
numeric_attribute: String (0.0)
bigdecimal_attribute: String (0.0)
float_attribute: String (0.0)
double_attribute: Real (0.0)
real_attribute: String (0.0)
date_attribute: DateTime (0.0)
time_attribute: String (0.0)
timestamp_attribute: String (0.0)
object_attribute: String (0.0)
Most new types are converted into strings by now. I think I
could sponsor the GDAL development if someone writes the
mapping between OpenJUMP datatypes and GDAL datatypes. I am
not sure what the GDAL datatypes are but the following list
might be right:
1. boolean (GDAL >= 2.0)
2. character(field_length). By default, field_length=1.
3. float(field_length)
4. numeric(field_length, field_precision)
5. smallint(field_length) : 16 bit signed integer (GDAL >= 2.0)
6. integer(field_length)
7. bigint(field_length), 64 bit integer, extension to SQL92
(GDAL >= 2.0)
8. date(field_length)
9. time(field_length)
10. timestamp(field_length)
Obviously 'string', and 'real' are supported and then I
believe that there is support for these three:
'integer list', 'double list' and 'string list'.
However, I have not yet realized if we need all these data
types and how generally they are supported in the surrounding
GIS and computing world.
-Jukka Rahkonen-
------------------------------------------------------------------------
Michaël Michaud wrote:
Hi,
I managed new attribute types introduced by Ede in shapefile
and jml drivers (BOOLEAN and LONG are managed as "Logical" and
"Numeric,21,0", and other types are managed as one of the
former types).
Please use and test these new types, especially with shapefiles.
Next step is to introduce these types in PostGIS database driver.
Michaël
-------- Message transféré --------
*Sujet : *
[jump-pilot:code] [r4337] - michaudm: Manage new attribute
types BOOLEAN and LONG in jml and shp drivers (also manage
other datatypes so that they can be written as one of the
well-known datatype instead of throwing an error message)
*Date : *
Sat, 14 Mar 2015 16:55:29 +0000
*De : *
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net>
*Répondre à : *
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net>
*Pour : *
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net>
Manage new attribute types BOOLEAN and LONG in jml and shp
drivers
(also manage other datatypes so that they can be written as
one of the well-known datatype instead of throwing an error
message)
http://sourceforge.net/p/jump-pilot/code/4337/<http://sourceforge.net/p/jump-pilot/code/4337>
------------------------------------------------------------------------
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/jump-pilot/code/<https://sourceforge.net/p/jump-pilot/code>
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/<https://sourceforge.net/auth/subscriptions>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub
for all
things parallel software development, from weekly thought leadership
blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for
all
things parallel software development, from weekly thought leadership blogs
to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
