Re: [base] Bugs in migration process
Thanks for testing the migration with Postgres First, here is some general information about Postgres and the migration: 1. We don't have access to a BASE 1 server running on Postgres. Thus we are not able to test or develop for it. 2. With BASE 2 we only support Postgres 8.2 and up. This means that we will not be able to fix anything that is related to Postgres in the migration. The only exception is if the error happens on the BASE 2 side which is not caused by a too old Postgres version. What we can do is to include patches (made by others) in the main distribution provided that they work with MySQL as well. Since users with BASE 1 on Postgres probably are not able to test with MySQL we will do the tests with MySQL here. See below for comments on each specific issue. Pawel Sztromwasser wrote: Hi, We are doing some preliminary migrations on testing databases to test if everything works fine with the migration script. We use Postgres 7.3.9 on both, Base1 and Base2 installations, so we will test how it works with this db server. Except changes required by Postgres in /migration-queries.sql/ file I found that in method /net.sf.basedb.clients.migrate.Migrater.verifySchemaVersion(Migrater.java:531)/ query checking schema version has to be changed (double quotes are necessary). Without this migration from Postgres database won't even start. We can externalize this query to the migration-queries.sql file. It should be possible to have it included in the 2.5 release next week. The second thing is transferring wizzzards. In our Base1 database those with /formatType=reportermap/, have column and row numbers in /mapping/ property proceeded by /_r_x /and/ _r_y/ respectively, not by /numCol/ and /numRow/ as WizzzardTransfer class expects. That causes the ReporterMapImporter plugin configuration to fail. I don't know if the mapping format has changed, but if the old one is still in use, the migration could support it. BASE 2 supports the same mappings as BASE 1 (and some more). I think it should be backwards slashes, eg. \_r_x\, etc. I am not sure what you mean with that the WizzzardTransfer expects \numCol\. The migration doesn't care about the mapping expressions. It should transfer them exactly as they are. Please post any error messages/stack traces, etc. And the last thing concerning only older Postgres server versions ( 8.0 according to http://archives.postgresql.org/pgsql-jdbc/2005-11/msg00127.php) is that they don't support JDBC's PreparedStatement.getParameterMetaData method. Luckily this method is not used very often and only minor changes in code (Transfer.java, SampleTissueTransfer.java, UserTransfer.java) were made to make migration run without this method. Users with 8.* Postgres versions shouldn't have this problem. I'm sorry but we are not able to help you with this. If your fixes also works with Postgres 8.x and MySQL we can include them in the main distribution. We are just about to start with the migration tests for the 2.5 release. If you want the changes tested (and possible included) in the 2.5 release I really need the code today. This brings up the discussion about how to submit a patch. There is some information on the developer page (I want to contribute): http://base.thep.lu.se/wiki/DeveloperInformation The instructions require that you have knowledge of Subversion and has checked out a local copy of the trunk. /Nicklas - SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Bugs in migration process
Nicklas Nordborg wrote:
Thanks for testing the migration with Postgres
First, here is some general information about Postgres and the migration:
1. We don't have access to a BASE 1 server running on Postgres. Thus we
are not able to test or develop for it.
2. With BASE 2 we only support Postgres 8.2 and up.
I remember that from the last time when JobAgents couldn't work
simultanously, but anyway, somebody might find it useful.
This means that we will not be able to fix anything that is related to
Postgres in the migration. The only exception is if the error happens on
the BASE 2 side which is not caused by a too old Postgres version.
What we can do is to include patches (made by others) in the main
distribution provided that they work with MySQL as well. Since users
with BASE 1 on Postgres probably are not able to test with MySQL we will
do the tests with MySQL here. See below for comments on each specific issue.
Pawel Sztromwasser wrote:
Hi,
We are doing some preliminary migrations on testing databases to test if
everything works fine with the migration script. We use Postgres 7.3.9
on both, Base1 and Base2 installations, so we will test how it works
with this db server. Except changes required by Postgres in
/migration-queries.sql/ file I found that in method
/net.sf.basedb.clients.migrate.Migrater.verifySchemaVersion(Migrater.java:531)/
query checking schema version has to be changed (double quotes are
necessary). Without this migration from Postgres database won't even start.
We can externalize this query to the migration-queries.sql file. It
should be possible to have it included in the 2.5 release next week.
Sounds good.
The second thing is transferring wizzzards. In our Base1 database those
with /formatType=reportermap/, have column and row numbers in /mapping/
property proceeded by /_r_x /and/ _r_y/ respectively, not by /numCol/
and /numRow/ as WizzzardTransfer class expects. That causes the
ReporterMapImporter plugin configuration to fail. I don't know if the
mapping format has changed, but if the old one is still in use, the
migration could support it.
BASE 2 supports the same mappings as BASE 1 (and some more). I think it
should be backwards slashes, eg. \_r_x\, etc. I am not sure what you
mean with that the WizzzardTransfer expects \numCol\. The migration
doesn't care about the mapping expressions. It should transfer them
exactly as they are. Please post any error messages/stack traces, etc.
I am sorry, the backslashes were added by Thunderbird when changing from
italic to text format.
The wizzard transfer works like this (when comes to mapping):
1) it parses mapping string from Base1 database using regular expression
looking for (key:value) pairs separated by comma (example part of a
mapping string:
_r_block:1,_r_x:\Column\,_r_y:\Row\,reporterId:\Probe_ID\, ...)
2) it puts the (key,value) pairs into HashMap
3) and uses this map when configuring import plugin. When getting those
values, it looks for certain keys ('numCol' and 'numRow' are among them,
explicitly coded around lines 130 and 140 in WizzzardTransfer.java).
When there is no value in HashMap it returns null, so null is passed to
plugin configuration. Then it complains:
10:58:47,690 DEBUG WizzzardTransfer:281 - Setting parameter values for a
reportermap-FormatType
10:58:47,690 DEBUG WizzzardTransfer:314 -
net.sf.basedb.core.InvalidUseOfNullException: Invalid use of null. The
'columnColumnMapping' value cannot be set to null.
10:58:47,690 DEBUG WizzzardTransfer:314 -
net.sf.basedb.core.InvalidUseOfNullException: Invalid use of null. The
'rowColumnMapping' value cannot be set to null.
10:58:47,690 WARN WizzzardTransfer:321 - The wizzard [11:Agilent test
map format w/o block] could not be transfered
It is not checking for _r_x, nor _r_y values anywhere. Probably if
_r_x/y values were substituted by numCol/Row in db it would work fine.
But it is easier to add and IF clause checking if
HashMap.get('numCol/Row') doesn't return null then alter database
records before migration:)
And the last thing concerning only older Postgres server versions ( 8.0
according to
http://archives.postgresql.org/pgsql-jdbc/2005-11/msg00127.php) is that
they don't support JDBC's PreparedStatement.getParameterMetaData method.
Luckily this method is not used very often and only minor changes in
code (Transfer.java, SampleTissueTransfer.java, UserTransfer.java) were
made to make migration run without this method. Users with 8.* Postgres
versions shouldn't have this problem.
I'm sorry but we are not able to help you with this. If your fixes also
works with Postgres 8.x and MySQL we can include them in the main
distribution. We are just about to start with the migration tests for
the 2.5 release. If you want the changes tested (and possible included)
in the 2.5 release I really need the code today.
My
Re: [base] Bugs in migration process
Pawel Sztromwasser wrote:
The second thing is transferring wizzzards. In our Base1 database those
with /formatType=reportermap/, have column and row numbers in /mapping/
property proceeded by /_r_x /and/ _r_y/ respectively, not by /numCol/
and /numRow/ as WizzzardTransfer class expects. That causes the
ReporterMapImporter plugin configuration to fail. I don't know if the
mapping format has changed, but if the old one is still in use, the
migration could support it.
BASE 2 supports the same mappings as BASE 1 (and some more). I think it
should be backwards slashes, eg. \_r_x\, etc. I am not sure what you
mean with that the WizzzardTransfer expects \numCol\. The migration
doesn't care about the mapping expressions. It should transfer them
exactly as they are. Please post any error messages/stack traces, etc.
I am sorry, the backslashes were added by Thunderbird when changing from
italic to text format.
The wizzard transfer works like this (when comes to mapping):
1) it parses mapping string from Base1 database using regular expression
looking for (key:value) pairs separated by comma (example part of a
mapping string:
_r_block:1,_r_x:\Column\,_r_y:\Row\,reporterId:\Probe_ID\, ...)
2) it puts the (key,value) pairs into HashMap
3) and uses this map when configuring import plugin. When getting those
values, it looks for certain keys ('numCol' and 'numRow' are among them,
explicitly coded around lines 130 and 140 in WizzzardTransfer.java).
When there is no value in HashMap it returns null, so null is passed to
plugin configuration. Then it complains:
Hmm... it seems like the code transfering 'reportermap' file formats is
not correctly coded. It doesn't look for any coordinates mappings at
all. The 'numCol' and 'numRow' you are referring to are part of the
'result' file formats, not 'reportermap' file formats.
On the good side is that this doesn't stop the migration. The only
problem is that the file formats for reporter maps are not transfered.
My changes should work with MySql with no problems, but they are not
'nice-coded'.
Failing method PreparedStatement.getParameterMetaData is used in three
places in code :
- run() : Transfer.java
- runUnbatched() : Transfer.java
- run() : SampleTissueTransfer.java
Each time only for getting parameterCount from statement query (number
of question marks in query in migration-queries.sql file). Only two
queries have this value != 2 and none of the classes using those queries
uses methods mentioned above. So what I did was just commenting this:
//int parameters = ps.getParameterMetaData().getParameterCount();
and adding:
int parameters = 2;
in those methods. Not very nice-coded, but little change that makes it
work. I am sure you will find better way to code that. I did not change
the number of question marks in none of the queries so it should work
with MySql as well.
We will not do anything about the problem. Since Postgres has fixed it
in a later version our policy is that users should upgrade. The
hard-coded approach is not solid enough to make it into the main
distribution.
/Nicklas
-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell. From the desktop to the data center, Linux is going
mainstream. Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
The BASE general discussion mailing list
basedb-users@lists.sourceforge.net
unsubscribe: send a mail with subject unsubscribe to
[EMAIL PROTECTED]
Re: [base] Bugs in migration process
Nicklas Nordborg wrote:
Pawel Sztromwasser wrote:
The second thing is transferring wizzzards. In our Base1 database those
with /formatType=reportermap/, have column and row numbers in /mapping/
property proceeded by /_r_x /and/ _r_y/ respectively, not by /numCol/
and /numRow/ as WizzzardTransfer class expects. That causes the
ReporterMapImporter plugin configuration to fail. I don't know if the
mapping format has changed, but if the old one is still in use, the
migration could support it.
BASE 2 supports the same mappings as BASE 1 (and some more). I think it
should be backwards slashes, eg. \_r_x\, etc. I am not sure what you
mean with that the WizzzardTransfer expects \numCol\. The migration
doesn't care about the mapping expressions. It should transfer them
exactly as they are. Please post any error messages/stack traces, etc.
I am sorry, the backslashes were added by Thunderbird when changing from
italic to text format.
The wizzard transfer works like this (when comes to mapping):
1) it parses mapping string from Base1 database using regular expression
looking for (key:value) pairs separated by comma (example part of a
mapping string:
_r_block:1,_r_x:\Column\,_r_y:\Row\,reporterId:\Probe_ID\, ...)
2) it puts the (key,value) pairs into HashMap
3) and uses this map when configuring import plugin. When getting those
values, it looks for certain keys ('numCol' and 'numRow' are among them,
explicitly coded around lines 130 and 140 in WizzzardTransfer.java).
When there is no value in HashMap it returns null, so null is passed to
plugin configuration. Then it complains:
Hmm... it seems like the code transfering 'reportermap' file formats is
not correctly coded. It doesn't look for any coordinates mappings at
all. The 'numCol' and 'numRow' you are referring to are part of the
'result' file formats, not 'reportermap' file formats.
Good point. I missed that IF checking for formatType. Are you planning
to fix that in 2.5?
On the good side is that this doesn't stop the migration. The only
problem is that the file formats for reporter maps are not transfered.
My changes should work with MySql with no problems, but they are not
'nice-coded'.
Failing method PreparedStatement.getParameterMetaData is used in three
places in code :
- run() : Transfer.java
- runUnbatched() : Transfer.java
- run() : SampleTissueTransfer.java
Each time only for getting parameterCount from statement query (number
of question marks in query in migration-queries.sql file). Only two
queries have this value != 2 and none of the classes using those queries
uses methods mentioned above. So what I did was just commenting this:
//int parameters = ps.getParameterMetaData().getParameterCount();
and adding:
int parameters = 2;
in those methods. Not very nice-coded, but little change that makes it
work. I am sure you will find better way to code that. I did not change
the number of question marks in none of the queries so it should work
with MySql as well.
We will not do anything about the problem. Since Postgres has fixed it
in a later version our policy is that users should upgrade. The
hard-coded approach is not solid enough to make it into the main
distribution.
It's OK with me. As long there is a simple workaround nobody will be
upset I suppose.
/Nicklas
-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell. From the desktop to the data center, Linux is going
mainstream. Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
The BASE general discussion mailing list
basedb-users@lists.sourceforge.net
unsubscribe: send a mail with subject unsubscribe to
[EMAIL PROTECTED]
-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell. From the desktop to the data center, Linux is going
mainstream. Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
The BASE general discussion mailing list
basedb-users@lists.sourceforge.net
unsubscribe: send a mail with subject unsubscribe to
[EMAIL PROTECTED]
