Hi Ede, The problem is the opposite. Changing schema is conscious choice but when user saves data to an existing shapefile the changed schema may not be saved because old field definitions Number(X.Y) and maybe also Character(X) are cached. DBF has only Number and Double for numbers http://www.clicketyclick.dk/databases/xbase/format/data_types.html and all the OJ data types must be mapped into Number(X.Y) format. What happens if user changes existing Number(10.0) that old OJ versions read as Integer but new one as Long into Integer, Tiny, or Small? Will the number format change or not? Should it change?
-Jukka Rahkonen- ________________________________________ Lähettäjä: edgar.sol...@web.de <edgar.sol...@web.de> Lähetetty: 18. maaliskuuta 2015 0:40 Vastaanottaja: OpenJump develop and use Aihe: Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in jml and shp drivers well, changing schema is a conscious choice. but anyway, how about warning on save that the column definitions have changed? but why the worries? i understand that this does not deviate from the standard or did i get wrong? ..ede On 17.03.2015 23:29, Rahkonen Jukka (MML) wrote: > Hi, > > > Preserving field sizes of existing shapefiles makes sense (well, in case of > Save as... I am not sure, perhaps even then) and probably did not leave to > any trouble when we had only string, integer, and double types. But now if > user wants to edit schema and change long->integer->small->tiny etc. and save > the edited shapefile the result may be a shapefile with different field sizes > than what user may believe. Users are rather helpless because they can never > see the field sizes of shapefiles which are opened or which will be created. > > > I do not really know what would be the best behaviour but I think that before > we make the next stable release we should find some logical or at least > documented solution. > > > -Jukka- > > ________________________________ > Michaël Michaud wrote: > >> Hi Jukka, > >> Yes caching the field size of a shapefile is a five years old patch from >> Larry Becker. > I think it is quite useful as it makes possible to preserve the schema of > shapefiles > created outside OpenJUMP. > > Michaël > > Le 17/03/2015 14:08, Rahkonen Jukka (MML) a écrit : > Hi, > > Please stop this investigation line, there is a bug but in another place. I > was testing by saving new results every time to same existing shapefile. When > I made the first save I had value “1234567890” in “integer_attribute”. > Therefore it received format Number(10.0). > > The bug is that if I edit the field in OpenJUMP to contain value “12345678” > and save the result with the same shapefile name, the field is still > Number(10.0). If I save it with a new name the field will be correctly > Number(9.0). > > Conclusion: OpenJUMP must be keeping some field definitions cached and > reused, or just updating the data of dbf file but not headers of something > else as funny. > > -Jukka- > > Lähettäjä: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] > Lähetetty: 17. maaliskuuta 2015 14:57 > Vastaanottaja: > jump-pilot-devel@lists.sourceforge.net<mailto:jump-pilot-devel@lists.sourceforge.net> > Aihe: Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in jml and > shp drivers > > > Hi, > > > > Find attached. Data contains: > > <property name="integer_attribute">1234567</property> > > but OpenOffice shows that the dbf field is defined as "integer_at,N,10,0" > > > > However, I made a quick test with one point and only one attribute and it > worked as you describe. Could it be that existence of some other fields may > lead to wrong result from your field length test? > > > > -Jukka- > > > > > > ________________________________ > Lähettäjä: Michaël Michaud > <m.michael.mich...@orange.fr<mailto:m.michael.mich...@orange.fr>> > Lähetetty: 17. maaliskuuta 2015 14:36 > Vastaanottaja: > jump-pilot-devel@lists.sourceforge.net<mailto:jump-pilot-devel@lists.sourceforge.net> > Aihe: Re: [JPP-Devel] Manage new attribute types BOOLEAN and LONG in jml and > shp drivers > > Jukka, > > In last revision, OJ checks the dataset first > If there is no value larger than 999 999 999 or lesser than -99 999 999, it > should use N(9,0) > but if the integer attribute contains values needing more than 9 digits, it > will choose the > field width accordingly (10 or 11). > The same should happen for longs (18 by default and 19 or 20 or 21 if needed). > I made a small test to confirm that a integer attribute containing a simgle > value '123456789' > is saved as N(9,0) and read back as an Integer. > If you have a counter example, please, send it to me in jml format. > > Michaël > > Le 17/03/2015 11:34, Rahkonen Jukka (MML) a écrit : > Hi, > > This must be some programming language magic, but even I can see in your code > for r4341 and I on testing with corresponding snapshot: > > else if (maxlength <= 9) fields[f] = new DbfFieldDef(columnName, 'N', 9, 0); > > OJ is still creating Integer field as (10.0). As a result a roundtrip with > OpenJUMP is now changing data type write into shp -> open back with OJ -> > what used to be integer is now Long. > > In the same way, Long is saved as (19.0) even the plan was to make it (18.0). > I have verified this with OpenOffice Calc which is nice because I can also > edit the format by hand, and with GDAL: > > This is how GDAL sees the fields now: > string_att: String (6.0) > char_attri: String (14.0) > varchar_at: String (17.0) > longvarcha: String (21.0) > text_attri: String (14.0) > boolean_at: String (1.0) > bit_attrib: String (1.0) > smallint_a: Integer (6.0) > tinyint_at: Integer (6.0) > integer_at: Integer64 (10.0) > long_attri: Real (19.0) > bigint_att: Real (19.0) > decimal_at: Real (33.16) > numeric_at: Real (33.16) > bigdecimal: Real (33.16) > float_attr: Real (33.16) > double_att: Real (33.16) > real_attri: Real (33.16) > date_attri: Date (10.0) > time_attri: Date (10.0) > timestamp_: Date (10.0) > object_att: String (1.0) > > -Jukka- > > > > Michaël Michaud wrote: > > Jukka, > > OK, I tried to implement it more or less as described in your previous mail, > > One other drawback with this change is that all previous shapefiles > containing integers and saved as N(11,0) will now be read back as longs. > > Let me now if someone think this can be a problem, > > Michaël > Le 16/03/2015 10:39, Rahkonen Jukka (MML) a écrit : > > Hi Michaël, > > > > My test file can now be successfully saved and read to/from shape and JML. > > I made also a test with GDAL-dev versions with the created shapefile > > > > ogrinfo -so -al datatype_test.shp > INFO: Open of `datatype_test.shp' > using driver `ESRI Shapefile' successful. > > Layer name: datatype_test > Geometry: Point > Feature Count: 1 > Extent: (310.000000, 406.000000) - (310.000000, 406.000000) > Layer SRS WKT: > (unknown) > string_att: String (6.0) > char_attri: String (14.0) > varchar_at: String (17.0) > longvarcha: String (21.0) > text_attri: String (14.0) > boolean_at: String (1.0) > bit_attrib: String (1.0) > smallint_a: Integer64 (11.0) > tinyint_at: Integer64 (11.0) > integer_at: Integer64 (11.0) > long_attri: Real (21.0) > bigint_att: Real (21.0) > decimal_at: Real (33.16) > numeric_at: Real (33.16) > bigdecimal: Real (33.16) > float_attr: Real (33.16) > double_att: Real (33.16) > real_attri: Real (33.16) > date_attri: Date (10.0) > time_attri: Date (10.0) > timestamp_: Date (10.0) > object_att: String (1.0) > > > > Notice that all the short integers are interpreted as long integers > (Integer64) and the long ones as Reals. Perhaps you should consider to make > the numbers a little bit shorter? According to this ticket > > http://trac.osgeo.org/gdal/ticket/3615 ever a number with 10 digits (10.0) > can be too big as an Integer. I suppose that the biggest Integer is > 4,294,967,29. And numbers with 20 digits can be bigger than Long integers if > they are >18,446,744,073,709,551,615. > > > > GDAL shp driver > > http://www.gdal.org/drv_shapefile.html is behaving this way when it creates > numeric fields : > > • Integer fields without an explicit width are treated as width 9, > and extended to 10 or 11 if needed. > > • Integer64 fields without an explicit width are treated as width 18, > and extended to 19 or 20 if needed. > > I made some tests about what GDAL does in reading. It appears to reports > numbers only up to (9.0) as Integers and up to (18.0) as Long Integers. > > > > I wonder if it would be better to change the shapefile writer of OpenJUMP to > create Integer fields by default as (9.0) and change format into (10.0) only > if integer field contains values between 1,000,000,00 and 4,294,967,29. > Bigger values than the upper limit should not be accepted into integer field > because they are invalid everywhere. > > > > There seems to be another issue with Long integers in the shapefiles. Long > integers can need up to 20 digits but the standard dBase format has an 18 > digit limit for numbers > > http://www.clicketyclick.dk/databases/xbase/format/data_types.html.<http://www.clicketyclick.dk/databases/xbase/format/data_types.html> > Some version has extended that to 20 numbers. Because of best possible > interoperability I think that OpenJUMP should create the Long fields as > (18.0) by default and (19.0) or (20.0) only if needed. > > > > -Jukka Rahkonen- > > > > > > > > ________________________________ > Michaël Michaud wrote: > >> Hi Jukka, > >> Thank you for the test and sorry for the exceptions. > I just completed with BIGINT, TIME and NUMERIC. > Shapefile driver will not really handle all these types. I've just handled > long and boolean in a specific way. Other types are just mapped to old > types. > >> This is how new types are supposed to be converted to dbf then back to >> OpenJUMP : > > CHAR, VARCHAR, LONGVARCHAR, TEXT, STRING, OBJECT -> C -> STRING > FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL, BIGDECIMAL -> N(33,16) -> DOUBLE > TINYINT, SMALLINT, INTEGER -> N(11) -> INTEGER > LONG, BIGINT -> N(21) -> LONG > DATE, TIME, TIMESTAMP -> D -> DATE > BOOLEAN, BIT -> L -> BOOLEAN > > The only data types that I've sometimes missed are boolean and long. > That's why I tried to map them in a way that can preserve type information > when > you save to dbf and back. > For other data types, my main concern is just to make the drivers compatible > with > the UI. > > Michaël > Le 15/03/2015 18:26, Rahkonen Jukka (MML) a écrit : > > Hi, > > > > I made a test file with one point and one attribute of each selectable data > type. However, OpenJUMP is not totally ready for handling all those. > > > > Saving the JML file as shapefile stops to the following error: > > > > java.lang.Exception: ShapefileWriter: unsupported AttributeType found in > featurecollection. : BIGINT > at > com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:537) > at > com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:292) > at > com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:73) > at > com.vividsolutions.jump.workbench.datasource.AbstractSaveDatasetAsPlugIn.run(AbstractSaveDatasetAsPlugIn.java:28) > at > com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:152) > at java.lang.Thread.run(Unknown Source) > > > > Next I tried to edit the schema and remove the BIGINT attribute but it was > not so easy. Changes in the schema can be confirmed only after removing all > the attributes of the following data types first: > > > > CHAR > > VARCHAR > > LONGVARCHAR > > TEXT > > BOOLEAN > > BIT > > SMALLINT > > TINYINT > > LONG > > BIGINT > > DECIMAL > > NUMERIC > > BIGDECIMAL > > FLOAT > > REAL > > TIME > > TIMESTAMP > > > > The error if any of the above data types appears in the schema is like: > > > > com.vividsolutions.jts.util.AssertionFailedException: Should never reach > here: VARCHAR > at > com.vividsolutions.jts.util.Assert.shouldNeverReachHere(Assert.java:122) > at > com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:557) > at > com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.convert(ViewSchemaPlugIn.java:286) > at > com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.applyChanges(ViewSchemaPlugIn.java:164) > at > com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn.access$300(ViewSchemaPlugIn.java:76) > at > com.vividsolutions.jump.workbench.ui.plugin.ViewSchemaPlugIn$EditSchemaFrame$3.actionPerformed(ViewSchemaPlugIn.java:695) > at > com.vividsolutions.jump.workbench.ui.SchemaPanel.fireActionPerformed(SchemaPanel.java:686) > at > com.vividsolutions.jump.workbench.ui.SchemaPanel.applyButton_actionPerformed(SchemaPanel.java:676) > at > com.vividsolutions.jump.workbench.ui.SchemaPanel$14.actionPerformed(SchemaPanel.java:447) > at javax.swing.AbstractButton.fireActionPerformed(Unknown Source) > at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source) > at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source) > at javax.swing.DefaultButtonModel.setPressed(Unknown Source) > at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown > Source) > at java.awt.Component.processMouseEvent(Unknown Source) > at javax.swing.JComponent.processMouseEvent(Unknown Source) > at java.awt.Component.processEvent(Unknown Source) > at java.awt.Container.processEvent(Unknown Source) > at java.awt.Component.dispatchEventImpl(Unknown Source) > at java.awt.Container.dispatchEventImpl(Unknown Source) > at java.awt.Component.dispatchEvent(Unknown Source) > at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source) > at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source) > at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source) > at java.awt.Container.dispatchEventImpl(Unknown Source) > at java.awt.Window.dispatchEventImpl(Unknown Source) > at java.awt.Component.dispatchEvent(Unknown Source) > at java.awt.EventQueue.dispatchEventImpl(Unknown Source) > at java.awt.EventQueue.access$200(Unknown Source) > at java.awt.EventQueue$3.run(Unknown Source) > at java.awt.EventQueue$3.run(Unknown Source) > at java.security.AccessController.doPrivileged(Native Method) > at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown > Source) > at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown > Source) > at java.awt.EventQueue$4.run(Unknown Source) > at java.awt.EventQueue$4.run(Unknown Source) > at java.security.AccessController.doPrivileged(Native Method) > at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown > Source) > at java.awt.EventQueue.dispatchEvent(Unknown Source) > at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) > at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) > at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) > at java.awt.EventDispatchThread.pumpEvents(Unknown Source) > at java.awt.EventDispatchThread.pumpEvents(Unknown Source) > at java.awt.EventDispatchThread.run(Unknown Source) > > > > Here is how GDAL understands our new attribute types: > > > > ogrinfo datatype_test.jml -al -so > Had to open data source read-only. > INFO: Open of `datatype_test.jml' > using driver `JML' successful. > > Layer name: dttest > Geometry: Unknown (any) > Feature Count: 1 > Extent: (310.000000, 406.000000) - (310.000000, 406.000000) > Layer SRS WKT: > (unknown) > string_attribute: String (0.0) > char_attribute: String (0.0) > varchar_attribute: String (0.0) > longvarchar_attribute: String (0.0) > text_attribute: String (0.0) > boolean_attribute: String (0.0) > bit_attribute: String (0.0) > smallint_attribute: String (0.0) > tinyint_attribute: String (0.0) > integer_attribute: Integer (0.0) > long_attribute: String (0.0) > bigint_attribute: String (0.0) > decimal_attribute: String (0.0) > numeric_attribute: String (0.0) > bigdecimal_attribute: String (0.0) > float_attribute: String (0.0) > double_attribute: Real (0.0) > real_attribute: String (0.0) > date_attribute: DateTime (0.0) > time_attribute: String (0.0) > timestamp_attribute: String (0.0) > object_attribute: String (0.0) > > > > Most new types are converted into strings by now. I think I could sponsor the > GDAL development if someone writes the mapping between OpenJUMP datatypes and > GDAL datatypes. I am not sure what the GDAL datatypes are but the following > list might be right: > > 1. boolean (GDAL >= 2.0) > 2. character(field_length). By default, field_length=1. > 3. float(field_length) > 4. numeric(field_length, field_precision) > 5. smallint(field_length) : 16 bit signed integer (GDAL >= 2.0) > 6. integer(field_length) > 7. bigint(field_length), 64 bit integer, extension to SQL92 (GDAL >= 2.0) > 8. date(field_length) > 9. time(field_length) > 10. timestamp(field_length) > > Obviously 'string', and 'real' are supported and then I believe that there is > support for these three: > > 'integer list', 'double list' and 'string list'. > > > > However, I have not yet realized if we need all these data types and how > generally they are supported in the surrounding GIS and computing world. > > > > -Jukka Rahkonen- > > > > > > > > > > ________________________________ > Michaël Michaud wrote: > > Hi, > > I managed new attribute types introduced by Ede in shapefile and jml drivers > (BOOLEAN and LONG are managed as "Logical" and "Numeric,21,0", and other > types are managed as one of the former types). > > Please use and test these new types, especially with shapefiles. > > Next step is to introduce these types in PostGIS database driver. > > Michaël > > -------- Message transféré -------- > Sujet : > > [jump-pilot:code] [r4337] - michaudm: Manage new attribute types BOOLEAN and > LONG in jml and shp drivers (also manage other datatypes so that they can be > written as one of the well-known datatype instead of throwing an error > message) > > Date : > > Sat, 14 Mar 2015 16:55:29 +0000 > > De : > > Repository The JUMP Pilot Project code > <nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net> > > Répondre à : > > Repository The JUMP Pilot Project code > <nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net> > > Pour : > > Repository The JUMP Pilot Project code > <nore...@code.jump-pilot.p.re.sf.net><mailto:nore...@code.jump-pilot.p.re.sf.net> > > > > Manage new attribute types BOOLEAN and LONG in jml and shp drivers > (also manage other datatypes so that they can be written as one of the > well-known datatype instead of throwing an error message) > http://sourceforge.net/p/jump-pilot/code/4337/<http://sourceforge.net/p/jump-pilot/code/4337> > > ________________________________ > > Sent from sourceforge.net because you indicated interest in > https://sourceforge.net/p/jump-pilot/code/<https://sourceforge.net/p/jump-pilot/code> > > To unsubscribe from further messages, please visit > https://sourceforge.net/auth/subscriptions/<https://sourceforge.net/auth/subscriptions> > > > > > ------------------------------------------------------------------------------ > > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > > by Intel and developed in partnership with Slashdot Media, is your hub for all > > things parallel software development, from weekly thought leadership blogs to > > news, videos, case studies, tutorials and more. Take a look and join the > > conversation now. http://goparallel.sourceforge.net/ > > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net> > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > ------------------------------------------------------------------------------ > > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > > by Intel and developed in partnership with Slashdot Media, is your hub for all > > things parallel software development, from weekly thought leadership blogs to > > news, videos, case studies, tutorials and more. Take a look and join the > > conversation now. http://goparallel.sourceforge.net/ > > > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net> > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > ------------------------------------------------------------------------------ > > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > > by Intel and developed in partnership with Slashdot Media, is your hub for all > > things parallel software development, from weekly thought leadership blogs to > > news, videos, case studies, tutorials and more. Take a look and join the > > conversation now. http://goparallel.sourceforge.net/ > > > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net> > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > by Intel and developed in partnership with Slashdot Media, is your hub for all > things parallel software development, from weekly thought leadership blogs to > news, videos, case studies, tutorials and more. Take a look and join the > conversation now. http://goparallel.sourceforge.net/ > > > > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net<mailto:Jump-pilot-devel@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > ------------------------------------------------------------------------------ > Dive into the World of Parallel Programming The Go Parallel Website, sponsored > by Intel and developed in partnership with Slashdot Media, is your hub for all > things parallel software development, from weekly thought leadership blogs to > news, videos, case studies, tutorials and more. Take a look and join the > conversation now. http://goparallel.sourceforge.net/ > > > > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
