Hi,
Please stop this investigation line, there is a bug but in another
place. I was testing by saving new results every time to same
existing shapefile. When I made the first save I had value
“1234567890” in “integer_attribute”. Therefore it received format
Number(10.0).
The bug is that if I edit the field in OpenJUMP to contain value
“12345678” and save the result with the same shapefile name, the field
is still Number(10.0). If I save it with a new name the field will be
correctly Number(9.0).
Conclusion: OpenJUMP must be keeping some field definitions cached and
reused, or just updating the data of dbf file but not headers of
something else as funny.
-Jukka-
*Lähettäjä:*Rahkonen Jukka (MML)
[mailto:jukka.rahko...@maanmittauslaitos.fi]
*Lähetetty:* 17. maaliskuuta 2015 14:57
*Vastaanottaja:* jump-pilot-devel@lists.sourceforge.net
*Aihe:* Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in
jml and shp drivers
Hi,
Find attached. Data contains:
<property name="integer_attribute">1234567</property>
but OpenOffice shows that the dbf field is defined as "integer_at,N,10,0"
However, I made a quick test with one point and only one attribute and
it worked as you describe. Could it be that existence of
some other fields may lead to wrong result from your field length test?
-Jukka-
------------------------------------------------------------------------
*Lähettäjä:*Michaël Michaud <m.michael.mich...@orange.fr
<mailto:m.michael.mich...@orange.fr>>
*Lähetetty:* 17. maaliskuuta 2015 14:36
*Vastaanottaja:* jump-pilot-devel@lists.sourceforge.net
<mailto:jump-pilot-devel@lists.sourceforge.net>
*Aihe:* Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in
jml and shp drivers
Jukka,
In last revision, OJ checks the dataset first
If there is no value larger than 999 999 999 or lesser than -99 999
999, it should use N(9,0)
but if the integer attribute contains values needing more than 9
digits, it will choose the
field width accordingly (10 or 11).
The same should happen for longs (18 by default and 19 or 20 or 21 if
needed).
I made a small test to confirm that a integer attribute containing a
simgle value '123456789'
is saved as N(9,0) and read back as an Integer.
If you have a counter example, please, send it to me in jml format.
Michaël
Le 17/03/2015 11:34, Rahkonen Jukka (MML) a écrit :
Hi,
This must be some programming language magic, but even I can see
in your code for r4341 and I on testing with corresponding snapshot:
else if (maxlength <= 9) fields[f] = new DbfFieldDef(columnName,
'N', 9, 0);
OJ is still creating Integer field as (10.0). As a result a
roundtrip with OpenJUMP is now changing data type write into shp
-> open back with OJ -> what used to be integer is now Long.
In the same way, Long is saved as (19.0) even the plan was to make
it (18.0). I have verified this with OpenOffice Calc which is
nice because I can also edit the format by hand, and with GDAL:
This is how GDAL sees the fields now:
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer (6.0)
tinyint_at: Integer (6.0)
integer_at: Integer64 (10.0)
long_attri: Real (19.0)
bigint_att: Real (19.0)
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
-Jukka-
MichaëlMichaud wrote:
Jukka,
OK, I tried to implement it more or less as described in your
previous mail,
One other drawback with this change is that all previous
shapefiles containing integers and saved as N(11,0) will now be
read back as longs.
Let me now if someone think this can be a problem,
Michaël
Le 16/03/2015 10:39, Rahkonen Jukka (MML) a écrit :
Hi Michaël,
My test file can now be successfully saved and read to/from
shape and JML.
I made also a test with GDAL-dev versions with the created
shapefile
ogrinfo -so -al datatype_test.shp
INFO: Open of `datatype_test.shp'
using driver `ESRI Shapefile' successful.
Layer name: datatype_test
Geometry: Point
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_att: String (6.0)
char_attri: String (14.0)
varchar_at: String (17.0)
longvarcha: String (21.0)
text_attri: String (14.0)
boolean_at: String (1.0)
bit_attrib: String (1.0)
smallint_a: Integer64 (11.0)
tinyint_at: Integer64 (11.0)
integer_at: Integer64 (11.0)
long_attri: Real (21.0)
bigint_att: Real (21.0)
decimal_at: Real (33.16)
numeric_at: Real (33.16)
bigdecimal: Real (33.16)
float_attr: Real (33.16)
double_att: Real (33.16)
real_attri: Real (33.16)
date_attri: Date (10.0)
time_attri: Date (10.0)
timestamp_: Date (10.0)
object_att: String (1.0)
Notice that all the short integers are interpreted as long
integers (Integer64) and the long ones as Reals. Perhaps you
should consider to make the numbers a little bit shorter?
According to this ticket
http://trac.osgeo.org/gdal/ticket/3615 ever a number with 10
digits (10.0) can be too big as an Integer. I suppose that the
biggest Integer is
4,294,967,29. And numbers with 20 digits can be bigger
than Long integers if they are >18,446,744,073,709,551,615.
GDAL shp driver
http://www.gdal.org/drv_shapefile.html is behaving this way
when it creates numeric fields :
·Integer fields without an explicit width are treated as width
9, and extended to 10 or 11 if needed.
·Integer64 fields without an explicit width are treated as
width 18, and extended to 19 or 20 if needed.
I made some tests about what GDAL does in reading. It appears
to reports numbers only up to (9.0) as Integers and up to
(18.0) as Long Integers.
I wonder if it would be better to change the shapefile writer
of OpenJUMP to create Integer fields by default as (9.0) and
change format into (10.0) only if integer field contains
values between 1,000,000,00 and 4,294,967,29. Bigger values
than the upper limit should not be accepted into integer field
because they are invalid everywhere.
There seems to be another issue with Long integers in the
shapefiles. Long integers can need up to 20 digits but the
standard dBase format has an 18 digit limit for numbers
http://www.clicketyclick.dk/databases/xbase/format/data_types.html.
<http://www.clicketyclick.dk/databases/xbase/format/data_types.html>
Some
version has extended that to 20 numbers. Because of best
possible interoperability I think that OpenJUMP should create
the Long fields as (18.0) by default and (19.0) or (20.0) only
if needed.
-Jukka Rahkonen-
------------------------------------------------------------------------
Michaël Michaud wrote:
> Hi Jukka,
> Thank you for the test and sorry for the exceptions.
I just completed with BIGINT, TIME and NUMERIC.
Shapefile driver will not really handle all these types. I've
just handled
long and boolean in a specific way. Other types are just
mapped to old
types.
> This is how new types are supposed to be converted to dbf
then back to OpenJUMP :
CHAR, VARCHAR, LONGVARCHAR, TEXT, STRING, OBJECT -> C
-> STRING
FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL, BIGDECIMAL -> N(33,16)
-> DOUBLE
TINYINT, SMALLINT, INTEGER -> N(11) -> INTEGER
LONG, BIGINT -> N(21) -> LONG
DATE, TIME, TIMESTAMP -> D -> DATE
BOOLEAN, BIT -> L -> BOOLEAN
The only data types that I've sometimes missed are boolean and
long.
That's why I tried to map them in a way that can preserve type
information when
you save to dbf and back.
For other data types, my main concern is just to make the
drivers compatible with
the UI.
Michaël
Le 15/03/2015 18:26, Rahkonen Jukka (MML) a écrit :
Hi,
I made a test file with one point and one attribute of
each selectable data type. However, OpenJUMP is not
totally ready for handling all those.
Saving the JML file as shapefile stops to the following error:
java.lang.Exception: ShapefileWriter: unsupported
AttributeType found in featurecollection. : BIGINT
at
com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:537)
at
com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:292)
at
com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:73)
at
com.vividsolutions.jump.workbench.datasource.AbstractSaveDatasetAsPlugIn.run(AbstractSaveDatasetAsPlugIn.java:28)
at
com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:152)
at java.lang.Thread.run(Unknown Source)
Next I tried to edit the schema and remove the BIGINT
attribute but it was not so easy. Changes in the schema
can be confirmed only after removing all the attributes of
the following data types first:
CHAR
VARCHAR
LONGVARCHAR
TEXT
BOOLEAN
BIT
SMALLINT
TINYINT
LONG
BIGINT
DECIMAL
NUMERIC
BIGDECIMAL
FLOAT
REAL
TIME
TIMESTAMP
The error if any of the above data types appears in the
schema is like:
com.vividsolutions.jts.util.AssertionFailedException:
Should never reach here: VARCHAR
at
com.vividsolutions.jts.util.Assert.shouldNeverReachHere(Assert.java:122)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:557)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:286)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.applyChanges(ViewSchemaPlugIn.java:164)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.access$300(ViewSchemaPlugIn.java:76)
at
com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn$EditSchemaFrame$3.actionPerformed(ViewSchemaPlugIn.java:695)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.fireActionPerformed(SchemaPanel.java:686)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel.applyButton_actionPerformed(SchemaPanel.java:676)
at
com.vividsolutions.jump.workbench.ui.SchemaPanel$14.actionPerformed(SchemaPanel.java:447)
at
javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at
javax.swing.AbstractButton$Handler.actionPerformed(Unknown
Source)
at
javax.swing.DefaultButtonModel.fireActionPerformed(Unknown
Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown
Source)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown
Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown
Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at
java.awt.LightweightDispatcher.retargetMouseEvent(Unknown
Source)
at
java.awt.LightweightDispatcher.processMouseEvent(Unknown
Source)
at
java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$200(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native
Method)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown
Source)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown
Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native
Method)
at
java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown
Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at
java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at
java.awt.EventDispatchThread.pumpEventsForFilter(Unknown
Source)
at
java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Here is how GDAL understands our new attribute types:
ogrinfo datatype_test.jml -al -so
Had to open data source read-only.
INFO: Open of `datatype_test.jml'
using driver `JML' successful.
Layer name: dttest
Geometry: Unknown (any)
Feature Count: 1
Extent: (310.000000, 406.000000) - (310.000000, 406.000000)
Layer SRS WKT:
(unknown)
string_attribute: String (0.0)
char_attribute: String (0.0)
varchar_attribute: String (0.0)
longvarchar_attribute: String (0.0)
text_attribute: String (0.0)
boolean_attribute: String (0.0)
bit_attribute: String (0.0)
smallint_attribute: String (0.0)
tinyint_attribute: String (0.0)
integer_attribute: Integer (0.0)
long_attribute: String (0.0)
bigint_attribute: String (0.0)
decimal_attribute: String (0.0)
numeric_attribute: String (0.0)
bigdecimal_attribute: String (0.0)
float_attribute: String (0.0)
double_attribute: Real (0.0)
real_attribute: String (0.0)
date_attribute: DateTime (0.0)
time_attribute: String (0.0)
timestamp_attribute: String (0.0)
object_attribute: String (0.0)
Most new types are converted into strings by now. I think
I could sponsor the GDAL development if someone writes the
mapping between OpenJUMP datatypes and GDAL datatypes. I
am not sure what the GDAL datatypes are but the following
list might be right:
1. boolean (GDAL >= 2.0)
2. character(field_length). By default, field_length=1.
3. float(field_length)
4. numeric(field_length, field_precision)
5. smallint(field_length) : 16 bit signed integer (GDAL
>= 2.0)
6. integer(field_length)
7. bigint(field_length), 64 bit integer, extension to
SQL92 (GDAL >= 2.0)
8. date(field_length)
9. time(field_length)
10. timestamp(field_length)
Obviously 'string', and 'real' are supported and then I
believe that there is support for these three:
'integer list', 'double list' and 'string list'.
However, I have not yet realized if we need all these data
types and how generally they are supported in the
surrounding GIS and computing world.
-Jukka Rahkonen-
------------------------------------------------------------------------
Michaël Michaud wrote:
Hi,
I managed new attribute types introduced by Ede in
shapefile and jml drivers (BOOLEAN and LONG are managed as
"Logical" and "Numeric,21,0", and other types are managed
as one of the former types).
Please use and test these new types, especially with
shapefiles.
Next step is to introduce these types in PostGIS database
driver.
Michaël
-------- Message transféré --------
*Sujet : *
[jump-pilot:code] [r4337] - michaudm: Manage new attribute
types BOOLEAN and LONG in jml and shp drivers (also manage
other datatypes so that they can be written as one of the
well-known datatype instead of throwing an error message)
*Date : *
Sat, 14 Mar 2015 16:55:29 +0000
*De : *
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net>
<mailto:nore...@code.jump-pilot.p.re.sf.net>
*Répondre à : *
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net>
<mailto:nore...@code.jump-pilot.p.re.sf.net>
*Pour : *
Repository The JUMP Pilot Project code
<nore...@code.jump-pilot.p.re.sf.net>
<mailto:nore...@code.jump-pilot.p.re.sf.net>
Manage new attribute types BOOLEAN and LONG in jml and shp
drivers
(also manage other datatypes so that they can be written
as one of the well-known datatype instead of throwing an
error message)
http://sourceforge.net/p/jump-pilot/code/4337/
<http://sourceforge.net/p/jump-pilot/code/4337>
------------------------------------------------------------------------
Sent from sourceforge.net because you indicated interest
in https://sourceforge.net/p/jump-pilot/code/
<https://sourceforge.net/p/jump-pilot/code>
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/
<https://sourceforge.net/auth/subscriptions>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel
Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your
hub for all
things parallel software development, from weekly thought
leadership blogs to
news, videos, case studies, tutorials and more. Take a look and
join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub
for all
things parallel software development, from weekly thought leadership
blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website,
sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for
all
things parallel software development, from weekly thought leadership blogs
to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now.http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
<mailto:Jump-pilot-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
