Hi,
Please stop this investigation line, there is a bug but in another place. I
was testing by saving new results every time to same existing shapefile. When I
made the first save I had value "1234567890" in "integer_attribute". Therefore
it received format Number(10.0).
The bug is that if I edit the field in OpenJUMP to contain value "12345678" and
save the result with the same shapefile name, the field is still Number(10.0).
If I save it with a new name the field will be correctly Number(9.0).
Conclusion: OpenJUMP must be keeping some field definitions cached and reused,
or just updating the data of dbf file but not headers of something else as
funny.
-Jukka-
Lähettäjä: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Lähetetty: 17. maaliskuuta 2015 14:57
Vastaanottaja: jump-pilot-devel@lists.sourceforge.net
Aihe: Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in jml and
shp drivers
Hi,
Find attached. Data contains:
<property name="integer_attribute">1234567</property>
but OpenOffice shows that the dbf field is defined as "integer_at,N,10,0"
However, I made a quick test with one point and only one attribute and it
worked as you describe. Could it be that existence of some other fields may
lead to wrong result from your field length test?
-Jukka-
________________________________
Lähettäjä: Michaël Michaud
<m.michael.mich...@orange.fr<mailto:m.michael.mich...@orange.fr>>
Lähetetty: 17. maaliskuuta 2015 14:36
Vastaanottaja:
jump-pilot-devel@lists.sourceforge.net<mailto:jump-pilot-devel@lists.sourceforge.net>
Aihe: Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in jml and
shp drivers
Jukka,
In last revision, OJ checks the dataset first
If there is no value larger than 999 999 999 or lesser than -99 999 999, it
should use N(9,0)
but if the integer attribute contains values needing more than 9 digits, it
will choose the
field width accordingly (10 or 11).
The same should happen for longs (18 by default and 19 or 20 or 21 if needed).
I made a small test to confirm that a integer attribute containing a simgle
value '123456789'
is saved as N(9,0) and read back as an Integer.
If you have a counter example, please, send it to me in jml format.
Michaël
Le 17/03/2015 11:34, Rahkonen Jukka (MML) a écrit :
Hi,
This must be some programming language magic, but even I can see in your code
for r4341 and I on testing with corresponding snapshot:
else if (maxlength <= 9) fields[f] = new DbfFieldDef(columnName, 'N', 9, 0);
OJ is still creating Integer field as (10.0). As a result a roundtrip with
OpenJUMP is now changing data type write into shp -> open back with OJ -> what
used to be integer is now Long.
In the same way, Long is saved as (19.0) even the plan was to make it (18.0).
I have verified this with OpenOffice Calc which is nice because I can also edit
the format by hand, and with GDAL:
This is how GDAL sees the fields now:
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer (6.0)
tinyint_at: Integer (6.0)
integer_at: Integer64 (10.0)
long_attri: Real (19.0)
bigint_att: Real (19.0)
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
-Jukka-
Michaël Michaud wrote:
Jukka,
OK, I tried to implement it more or less as described in your previous mail,
One other drawback with this change is that all previous shapefiles containing
integers and saved as N(11,0) will now be read back as longs.
Let me now if someone think this can be a problem,
Michaël
Le 16/03/2015 10:39, Rahkonen Jukka (MML) a écrit :
Hi Michaël,
My test file can now be successfully saved and read to/from shape and JML.
I made also a test with GDAL-dev versions with the created shapefile
ogrinfo -so -al datatype_test.shp
INFO: Open of `datatype_test.shp'
using driver `ESRI Shapefile' successful.
Layer name: datatype_test
Geometry: Point
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer64 (11.0)
tinyint_at: Integer64 (11.0)
integer_at: Integer64 (11.0)
long_attri: Real (21.0)
bigint_att: Real (21.0)
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
Notice that all the short integers are interpreted as long integers (Integer64)
and the long ones as Reals. Perhaps you should consider to make the numbers a
little bit shorter? According to this ticket
http://trac.osgeo.org/gdal/ticket/3615 ever a number with 10 digits (10.0) can
be too big as an Integer. I suppose that the biggest Integer is 4,294,967,29.
And numbers with 20 digits can be bigger than Long integers if they are
>18,446,744,073,709,551,615.
GDAL shp driver
http://www.gdal.org/drv_shapefile.html is behaving this way when it creates
numeric fields :
* Integer fields without an explicit width are treated as width 9, and
extended to 10 or 11 if needed.
* Integer64 fields without an explicit width are treated as width 18,
and extended to 19 or 20 if needed.
I made some tests about what GDAL does in reading. It appears to reports
numbers only up to (9.0) as Integers and up to (18.0) as Long Integers.
I wonder if it would be better to change the shapefile writer of OpenJUMP to
create Integer fields by default as (9.0) and change format into (10.0) only if
integer field contains values between 1,000,000,00 and 4,294,967,29. Bigger
values than the upper limit should not be accepted into integer field because
they are invalid everywhere.
There seems to be another issue with Long integers in the shapefiles. Long
integers can need up to 20 digits but the standard dBase format has an 18
digit limit for numbers
http://www.clicketyclick.dk/databases/xbase/format/data_types.html.<http://www.clicketyclick.dk/databases/xbase/format/data_types.html>
Some version has extended that to 20 numbers. Because of best possible
interoperability I think that OpenJUMP should create the Long fields as (18.0)
by default and (19.0) or (20.0) only if needed.
-Jukka Rahkonen-
________________________________
Michaël Michaud wrote:
> Hi Jukka,
> Thank you for the test and sorry for the exceptions.
I just completed with BIGINT, TIME and NUMERIC.
Shapefile driver will not really handle all these types. I've just handled
long and boolean in a specific way. Other types are just mapped to old
types.
> This is how new types are supposed to be converted to dbf then back to
> OpenJUMP :
CHAR, VARCHAR, LONGVARCHAR, TEXT, STRING, OBJECT -> C -> STRING
FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL, BIGDECIMAL -> N(33,16) -> DOUBLE
TINYINT, SMALLINT, INTEGER -> N(11) -> INTEGER
LONG, BIGINT -> N(21) -> LONG
DATE, TIME, TIMESTAMP -> D -> DATE
BOOLEAN, BIT -> L -> BOOLEAN
The only data types that I've sometimes missed are boolean and long.
That's why I tried to map them in a way that can preserve type information when
you save to dbf and back.
For other data types, my main concern is just to make the drivers compatible
with
the UI.
Michaël
Le 15/03/2015 18:26, Rahkonen Jukka (MML) a écrit :
Hi,
I made a test file with one point and one attribute of each selectable data
type. However, OpenJUMP is not totally ready for handling all those.
Saving the JML file as shapefile stops to the following error:
java.lang.Exception: ShapefileWriter: unsupported AttributeType found in
featurecollection. : BIGINT
at
com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:537)
at
com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:292)
at
com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:73)
at
com.vividsolutions.jump.workbench.datasource.AbstractSaveDatasetAsPlugIn.run(AbstractSaveDatasetAsPlugIn.java:28)
at
com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:152)
at java.lang.Thread.run(Unknown Source)
Next I tried to edit the schema and remove the BIGINT attribute but it was not
so easy. Changes in the schema can be confirmed only after removing all the
attributes of the following data types first:
CHAR
VARCHAR
LONGVARCHAR
TEXT
BOOLEAN
BIT
SMALLINT
TINYINT
LONG
BIGINT
DECIMAL
NUMERIC
BIGDECIMAL
FLOAT
REAL
TIME
TIMESTAMP
The error if any of the above data types appears in the schema is like:
com.vividsolutions.jts.util.AssertionFailedException: Should never reach here:
VARCHAR
at com.vividsolutions.jts.util.Assert.shouldNeverReachHere(Assert.java:122)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:557)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:286)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.applyChanges(ViewSchemaPlugIn.java:164)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.access$300(ViewSchemaPlugIn.java:76)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn$EditSchemaFrame$3.actionPerformed(ViewSchemaPlugIn.java:695)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.fireActionPerformed(SchemaPanel.java:686)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.applyButton_actionPerformed(SchemaPanel.java:676)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel$14.actionPerformed(SchemaPanel.java:447)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$200(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Here is how GDAL understands our new attribute types:
ogrinfo datatype_test.jml -al -so
Had to open data source read-only.
INFO: Open of `datatype_test.jml'
using driver `JML' successful.
Layer name: dttest
Geometry: Unknown (any)
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_attribute: String (0.0)
char_attribute: String (0.0)
varchar_attribute: String (0.0)
longvarchar_attribute: String (0.0)
text_attribute: String (0.0)
boolean_attribute: String (0.0)
bit_attribute: String (0.0)
smallint_attribute: String (0.0)
tinyint_attribute: String (0.0)
integer_attribute: Integer (0.0)
long_attribute: String (0.0)
bigint_attribute: String (0.0)
decimal_attribute: String (0.0)
numeric_attribute: String (0.0)
bigdecimal_attribute: String (0.0)
float_attribute: String (0.0)
double_attribute: Real (0.0)
real_attribute: String (0.0)
date_attribute: DateTime (0.0)
time_attribute: String (0.0)
timestamp_attribute: String (0.0)
object_attribute: String (0.0)
Most new types are converted into strings by now. I think I could sponsor the
GDAL development if someone writes the mapping between OpenJUMP datatypes and
GDAL datatypes. I am not sure what the GDAL datatypes are but the following
list might be right:
1. boolean (GDAL >= 2.0)
2. character(field_length). By default, field_length=1.
3. float(field_length)
4. numeric(field_length, field_precision)
5. smallint(field_length) : 16 bit signed integer (GDAL >= 2.0)
6. integer(field_length)
7. bigint(field_length), 64 bit integer, extension to SQL92 (GDAL >= 2.0)
8. date(field_length)
9. time(field_length)
10. timestamp(field_length)
Obviously 'string', and 'real' are supported and then I believe that there is
support for these three:
'integer list', 'double list' and 'string list'.
However, I have not yet realized if we need all these data types and how
generally they are supported in the surrounding GIS and computing world.
-Jukka Rahkonen-
________________________________
Michaël Michaud wrote:
Hi,
I managed new attribute types introduced by Ede in shapefile and jml drivers
(BOOLEAN and LONG are managed as "Logical" and "Numeric,21,0", and other types
are managed as one of the former types).
Please use and test these new types, especially with shapefiles.
Next step is to introduce these types in PostGIS database driver.
Michaël
-------- Message transféré --------
Sujet :
[jump-pilot:code] [r4337] - michaudm: Manage new attribute types BOOLEAN and
LONG in jml and shp drivers (also manage other datatypes so that they can be
written as one of the well-known datatype instead of throwing an error message)
Date :
Sat, 14 Mar 2015 16:55:29 +0000
De :
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net>
Répondre à :
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net>
Pour :
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net>
Manage new attribute types BOOLEAN and LONG in jml and shp drivers
(also manage other datatypes so that they can be written as one of the
well-known datatype instead of throwing an error message)
http://sourceforge.net/p/jump-pilot/code/4337/<http://sourceforge.net/p/jump-pilot/code/4337>
________________________________
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/jump-pilot/code/<https://sourceforge.net/p/jump-pilot/code>
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/<https://sourceforge.net/auth/subscriptions>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
