Re: [Oorexx-devel] Rainer are you still around?
Hello, any ideas before I start the real AIX builds?? Bye Rainer On 12.06.2012 09:37, Rainer Tammer wrote: Hello, I just found out, that URL was not correct... To be sure that there are no bigger problems with the 4.1.1 release I just build a 32 4.1.1 ooRexx and executed the test suite: ooTest Framework - Automated Test of the ooRexx Interpreter Interpreter: REXX-ooRexx_4.1.1(MT) 6.03 12 Jun 2012 Addressing Mode: 32 ooRexxUnit: 2.0.0_3.2.0ooTest: 1.0.0_4.0.0 Tests ran: 18993 Assertions: 575871 Failures:5 Errors: 0 Skipped files: 24 [failure] [20120612 09:30:26.373400] svn:r5224 Change date: 2009-09-30 16:17:43 +0200 Test: TEST_MANY_CHARS Class: CHAROUT.testGroup File: /daten/source/ooRexx_4_1_1/.../4.1.0/ooRexx/base/bif/CHAROUT.testGroup Line: 150 Failed: assertSame Expected: [[0], identityHash=100820088] Actual: [[43925504], identityHash=119673482] [failure] [20120612 09:30:28.722280] svn:r5224 Change date: 2009-09-30 16:17:43 +0200 Test: TEST_MANY_CHARS_BIN Class: CHAROUT.testGroup File: /daten/source/ooRexx_4_1_1/.../4.1.0/ooRexx/base/bif/CHAROUT.testGroup Line: 989 Failed: assertSame Expected: [[0], identityHash=100820088] Actual: [[43925504], identityHash=119678159] [failure] [20120612 09:30:32.042544] svn:r5224 Change date: 2009-09-30 16:17:43 +0200 Test: TEST_MANY_CHARS_NB Class: CHAROUT.testGroup File: /daten/source/ooRexx_4_1_1/.../4.1.0/ooRexx/base/bif/CHAROUT.testGroup Line: 577 Failed: assertSame Expected: [[0], identityHash=100820088] Actual: [[43925504], identityHash=119279073] [failure] [20120612 09:32:34.659518] svn:r6418 Change date: 2010-11-30 14:26:10 +0100 Test: TEST_9 Class: TIME.long.testGroup File: /daten/source/ooRexx_4_1_1/test/4.1.0/ooRexx/base/bif/TIME.testGroup Line: 1984 Failed: assertTrue Expected: [1] Actual: [[0], identityHash=100801275] Message: cb3a should be greater than or equal to 5, cb3a is: 4.999197 --- ignore test suite was executed as root - [failure] [20120612 09:33:28.795271] svn:r4966 Change date: 2009-07-21 01:44:03 +0200 Test: TEST_STATE_ERROR Class: Stream.testGroup File: /daten/source/ooRexx_4_1_1/.../4.1.0/ooRexx/base/class/Stream.testGroup Line: 1113 Failed: assertNotSame Expected: [[0], identityHash=100820088] \== [[0], identityHash=118062185] Actual: [[0], identityHash=100820088] == [[0], identityHash=118062185] Message: This test case must not be run as root --- ignore test suite was executed as root - Interpreter: REXX-ooRexx_4.1.1(MT) 6.03 12 Jun 2012 Addressing Mode: 32 ooRexxUnit: 2.0.0_3.2.0ooTest: 1.0.0_4.0.0 Tests ran: 18993 Assertions: 575871 Failures:5 Errors: 0 Skipped files: 24 File search:00:00:12.391469 Suite construction: 00:00:08.730104 Test execution: 00:07:32.269283 Total time: 00:07:58.476396 Are these known problems?? Bye Rainer On 08.06.2012 19:51, Mark Miesfeld wrote: Hi Rainer, We haven't heard from you in awhile. Are you still lurking around? If so, is it possible for you to build an AIX package for ooRexx 4.1.1? If it is not possible, for whatever reason, that's fine. I'm sure it is not a problem with any of the other developers. We all have real lives that we need to attend to. We'd still like to hear from you. Not being able to build a package should not prevent you from posting on any of the ooRexx lists. -- Mark Miesfeld -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
[Oorexx-devel] Fwd: Rainer are you still around?
Hi, I meant this to go to the dev list, not straight to Rainer. -- Mark Miesfeld -- Forwarded message -- From: Mark Miesfeld miesf...@gmail.com Date: Thu, Jun 14, 2012 at 8:10 AM Subject: Re: [Oorexx-devel] Rainer are you still around? To: Rainer Tammer tam...@tammer.net On Wed, Jun 13, 2012 at 11:11 PM, Rainer Tammer tam...@tammer.net wrote: Hi Rainer, First off, thanks a lot for doing this when your are so busy. Please don't feel any pressure to build the package until you have the time for it. any ideas before I start the real AIX builds?? My idea is that 4.1.1 has been released, so just use the release code in your build. If you persistently get errors running the test suite, you can open a bug. Having said that, I bet that this test failure is transient: [failure] [20120612 09:32:34.659518] svn: r6418 Change date: 2010-11-30 14:26:10 +0100 Test: TEST_9 Class: TIME.long.testGroup File: /daten/source/ooRexx_4_1_1/test/4.1.0/ooRexx/base/bif/TIME.testGroup Line: 1984 Failed: assertTrue Expected: [1] Actual: [[0], identityHash=100801275] Message: cb3a should be greater than or equal to 5, cb3a is: 4.999197 In other words if you run the test several times you won't see the failure every time. I don't think it indicates a bug in the interpreter code. I think it reflects that you can not expect the system clock to produce sub-millisecond accuracy in user space. That leaves the 3 CHAR test failures. I don't see those failures on Windows and I didn't see them on the several Linux systems I ran the test suite on at the time of the release. Here's Windows with a download of the 4.1.1 release: ooTest Framework - Automated Test of the ooRexx Interpreter Interpreter: REXX-ooRexx_4.1.1(MT) 6.03 16 May 2012 Addressing Mode: 64 ooRexxUnit: 2.0.0_3.2.0 ooTest: 1.0.0_4.0.0 Tests ran: 19257 Assertions: 576586 Failures: 1 Errors: 0 Skipped files: 1 [failure] [20120614 07:40:07.582000] svn: r3541 Change date: 2008-10-14 16:31:57 -0700 Test: TEST_READRECORDS05 Class: WindowsEventLog.testGroup File: C:\work.ooRexx\...\platform\windows\rxwinsys\WindowsEventLog.testGroup Line: 524 Failed: assertSame Expected: [[57413], identityHash=1099388258186] Actual: [[57412], identityHash=1099388258194] File search: 00:00:13.822000 Suite construction: 00:00:01.95 Test execution: 00:10:05.236000 Total time: 00:10:29.321000 The one failure shows that between the time the record events log was filled and the time of the second asset, 1 record was added to the event log. This is not unusual on a busy system and is perhaps a flaw in the test case, certainly not a bug in the interpreter or the extension.. I don't have a Linux system handy to re-test on right now. But, I didn't see those failures at the time of the release. So - I say build the AIX package with the release code. Open a bug if the test suite continues to fail on the CHAR tests. Just be sure you are using the actual release source: https://oorexx.svn.sourceforge.net/svnroot/oorexx/main/releases/4.1.1/trunk Thanks again for building the AIX packages Rainer -- Mark Miesfeld -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
Re: [Oorexx-devel] ooSQLite usage
As I saw that ooSQLite was updated in incubator, regarding the SQLite backup API,I downloaded the source and compiled it under windows 7 64bit. From the sample /testing/loadDatabase.rex, I made my own database loading rex. The program creates a sqlite database in memory, with 6 columns, read and parse a 181Mb file and creates SQL insert statements, then inserts the resulting records (101657 records) into the memory database and last step save the in memory database to disc. This programs elapsed time was around 61-65 seconds( including copy to disc) , compared to the previous ~17 MINUTES when using a disc based database only, with same data. Great improvment. By the way I also downloaded the latest SQLite source 3.7.13 and used that in the compile above. All works very well. /hex - Ursprungligt Meddelande: Från: hakan hexi...@users.sourceforge.net Till: Mark Miesfeld miesf...@gmail.com Kopia: Datum: fredag, 08 juni 2012 22:20 Ämne: Re: [Oorexx-devel] ooSQLite usage Well, tried with BEGIN TRANSACTION. COMMIT and cut the time by around 5 minutes, so when initial loading is done, it seems that inmemory database is much quickier, because of hardisc speed, if a possibility to save the inmemory database to harddisc is available. Looking forward to your implementation of the .backup (.dot ) commands in ooSQLite. Keep up your good work (as a side note I am very satisfied with ooDialog 4.2 excellent work, I love it, maybe as I now understand how it works (at least to some point) and how to use it together with ooRexx, OO is hard for an old mainframer :-) ) /hex - Ursprungligt Meddelande: Från: Mark Miesfeld miesf...@gmail.com Till: hexi...@users.sourceforge.net, Open Object Rexx Developer Mailing List oorexx-devel@lists.sourceforge.net Kopia: Datum: fredag, 08 juni 2012 21:13 Ämne: Re: [Oorexx-devel] ooSQLite usage Very cool. On Fri, Jun 8, 2012 at 11:07 AM, hakan hexi...@users.sourceforge.net wrote: After trying the ooSQLite package, I must say I am very impressed (well I am not a DB person, so the learning curve is step) but I find it useful for some of my ooREXX programs. The samples have been very helpful, I am now able to create, load and query a database/tables in ooREXX. One thing is performance, when creating a new database, I have read/parsed a file and then loaded it (92 000 rows) to a database/table , took around 17 minutes on a i3 2.13Ghz ! From my reading (but I'm also not a DB person) this is a problem with INSERTS with not using an explicit BEGIN TRANSACTION COMMIT. As I understand it, for every individual SQL statement executed, SQLite will implicitly add the BEGIN TRANSACTION / COMMIT if it is not already within a transaction. The SQLite engine only does about 4 or 5 inserts a second under this condition. If you add the explicit BEGIN TRANSACTION before the first insert and then end with COMMIT the speed should be more normal. Something like this: BEGIN TRANSACTION; INSERT INTO episodes VALUES(0, NULL, 'Good News Bad News'); INSERT INTO episodes VALUES(1, 1, 'Male Unbonding'); INSERT INTO episodes VALUES(2, 1, 'The Stake Out'); INSERT INTO episodes VALUES(3, 1, 'The Robbery'); ... INSERT INTO food_types VALUES(13, 'Seafood'); INSERT INTO food_types VALUES(14, 'Soup'); INSERT INTO food_types VALUES(15, 'Vegetables'); COMMIT; I guess it is COMMIT not END TRANSACTION. I first thought it was END TRANSACTION. The above is the raw SQL, I'm assuming you are wrapping the raw SQL in a prepared statement. I then tried to to create a ooSQLite database in memory instead of a file (ooSQLite supports this, I used :memory: instead of a filename) then loading the same amount of data took around 15-20 seconds, but there is one problem I can't save the in memory database to a file with ooSQLite, as I understand from SQLite website there exist an backup API that let you save an in memory database to disc. I have tried it with the ooSQLITE3.exe (cli interface) doing the same as the ooRexx program( except backup) above does, by opening a :memory: database then create a table, then use .read file with SQL inserts and last use the .backup file CLI command. The result is that with the same amont of rows (inserts) it took around 30 seconds to have the database copied to disc including the SQl inserts. So, Question is is it possible to implement the backup api (to be able to copy a in memory database to disc) in ooSQLite, maybe also to make inserts into a database from a file (like the .read file CLi command). ? I may have missed something regarding this! Yes, both of those things are next on my list. Or at least very near the top. I intend to support the backup API as the next coding I do. In addition, I've planned from the start to support the dot commands present in the command line shell in ooSQLite classes. The most important ones like .dump, .read, .schema, etc., first. Maybe not all of them, I'm
Re: [Oorexx-devel] ooSQLite usage
Good news. I'm still fooling with the backup object a little, but probably won't change anything at this point. I'm working on the doc for the ooSQLiteBackup object and the functional equivalents. My plan is to put up a new package this weekend that will have the online backup complete, along with doc finished for that portion. I have also updated my source with SQLite 3.7.13, I just haven't committed it yet. So, the next package will be based on 3.7.13. When the ooSQLite package reaches the release stage, my plan is update the package with the latest SQLite version when they do a release. Then, I will place the prebuilt binaries on SourceForge. This will allow users that are not comfortable with compiling and building to always have access to the latest SQLite version. At most, ooSQLite will lag by one day a new SQLite version. I'm trying to keep a running list of changes between package versions put on SourceForge, as I make a change, in the ReleaseNotes file. You may want to watch that file to be alerted to changes in the source that may effect you. -- Mark Miesfeld On Thu, Jun 14, 2012 at 12:22 PM, hakan hexi...@users.sourceforge.net wrote: As I saw that ooSQLite was updated in incubator, regarding the SQLite backup API,I downloaded the source and compiled it under windows 7 64bit. From the sample /testing/loadDatabase.rex, I made my own database loading rex. The program creates a sqlite database in memory, with 6 columns, read and parse a 181Mb file and creates SQL insert statements, then inserts the resulting records (101657 records) into the memory database and last step save the in memory database to disc. This programs elapsed time was around 61-65 seconds( including copy to disc) , compared to the previous ~17 MINUTES when using a disc based database only, with same data. Great improvment. By the way I also downloaded the latest SQLite source 3.7.13 and used that in the compile above. All works very well. /hex - Ursprungligt Meddelande: Från: hakan hexi...@users.sourceforge.net Till: Mark Miesfeld miesf...@gmail.com Kopia: Datum: fredag, 08 juni 2012 22:20 Ämne: Re: [Oorexx-devel] ooSQLite usage Well, tried with BEGIN TRANSACTION. COMMIT and cut the time by around 5 minutes, so when initial loading is done, it seems that inmemory database is much quickier, because of hardisc speed, if a possibility to save the inmemory database to harddisc is available. Looking forward to your implementation of the .backup (.dot ) commands in ooSQLite. Keep up your good work (as a side note I am very satisfied with ooDialog 4.2 excellent work, I love it, maybe as I now understand how it works (at least to some point) and how to use it together with ooRexx, OO is hard for an old mainframer :-) ) /hex - Ursprungligt Meddelande: Från: Mark Miesfeld miesf...@gmail.com Till: hexi...@users.sourceforge.net, Open Object Rexx Developer Mailing List oorexx-devel@lists.sourceforge.net Kopia: Datum: fredag, 08 juni 2012 21:13 Ämne: Re: [Oorexx-devel] ooSQLite usage Very cool. On Fri, Jun 8, 2012 at 11:07 AM, hakan hexi...@users.sourceforge.net wrote: After trying the ooSQLite package, I must say I am very impressed (well I am not a DB person, so the learning curve is step) but I find it useful for some of my ooREXX programs. The samples have been very helpful, I am now able to create, load and query a database/tables in ooREXX. One thing is performance, when creating a new database, I have read/parsed a file and then loaded it (92 000 rows) to a database/table , took around 17 minutes on a i3 2.13Ghz ! From my reading (but I'm also not a DB person) this is a problem with INSERTS with not using an explicit BEGIN TRANSACTION COMMIT. As I understand it, for every individual SQL statement executed, SQLite will implicitly add the BEGIN TRANSACTION / COMMIT if it is not already within a transaction. The SQLite engine only does about 4 or 5 inserts a second under this condition. If you add the explicit BEGIN TRANSACTION before the first insert and then end with COMMIT the speed should be more normal. Something like this: BEGIN TRANSACTION; INSERT INTO episodes VALUES(0, NULL, 'Good News Bad News'); INSERT INTO episodes VALUES(1, 1, 'Male Unbonding'); INSERT INTO episodes VALUES(2, 1, 'The Stake Out'); INSERT INTO episodes VALUES(3, 1, 'The Robbery'); ... INSERT INTO food_types VALUES(13, 'Seafood'); INSERT INTO food_types VALUES(14, 'Soup'); INSERT INTO food_types VALUES(15, 'Vegetables'); COMMIT; I guess it is COMMIT not END TRANSACTION. I first thought it was END TRANSACTION. The above is the raw SQL, I'm assuming you are wrapping the raw SQL in a prepared statement. I then tried to to create a ooSQLite database in memory instead of a file (ooSQLite supports this, I used :memory: instead of a filename) then loading the same amount of data took around
Re: [Oorexx-devel] Fwd: Rainer are you still around?
About the three failures TEST_MANY_CHARS. I have them under windows, when running the test from an USB drive. http://sourceforge.net/mailarchive/message.php?msg_id=26103047 Seems 200Mb is too big, and it remains around 42Mb not written... Disk full ? Jean-Louis 2012/6/14 Mark Miesfeld miesf...@gmail.com Hi, I meant this to go to the dev list, not straight to Rainer. -- Mark Miesfeld -- Forwarded message -- From: Mark Miesfeld miesf...@gmail.com Date: Thu, Jun 14, 2012 at 8:10 AM Subject: Re: [Oorexx-devel] Rainer are you still around? To: Rainer Tammer tam...@tammer.net On Wed, Jun 13, 2012 at 11:11 PM, Rainer Tammer tam...@tammer.net wrote: Hi Rainer, First off, thanks a lot for doing this when your are so busy. Please don't feel any pressure to build the package until you have the time for it. any ideas before I start the real AIX builds?? My idea is that 4.1.1 has been released, so just use the release code in your build. If you persistently get errors running the test suite, you can open a bug. Having said that, I bet that this test failure is transient: [failure] [20120612 09:32:34.659518] svn:r6418 Change date: 2010-11-30 14:26:10 +0100 Test: TEST_9 Class: TIME.long.testGroup File: /daten/source/ooRexx_4_1_1/test/4.1.0/ooRexx/base/bif/TIME.testGroup Line: 1984 Failed: assertTrue Expected: [1] Actual: [[0], identityHash=100801275] Message: cb3a should be greater than or equal to 5, cb3a is: 4.999197 In other words if you run the test several times you won't see the failure every time. I don't think it indicates a bug in the interpreter code. I think it reflects that you can not expect the system clock to produce sub-millisecond accuracy in user space. That leaves the 3 CHAR test failures. I don't see those failures on Windows and I didn't see them on the several Linux systems I ran the test suite on at the time of the release. Here's Windows with a download of the 4.1.1 release: ooTest Framework - Automated Test of the ooRexx Interpreter Interpreter: REXX-ooRexx_4.1.1(MT) 6.03 16 May 2012 Addressing Mode: 64 ooRexxUnit: 2.0.0_3.2.0ooTest: 1.0.0_4.0.0 Tests ran: 19257 Assertions: 576586 Failures:1 Errors: 0 Skipped files: 1 [failure] [20120614 07:40:07.582000] svn:r3541 Change date: 2008-10-14 16:31:57 -0700 Test: TEST_READRECORDS05 Class: WindowsEventLog.testGroup File: C:\work.ooRexx\...\platform\windows\rxwinsys\WindowsEventLog.testGroup Line: 524 Failed: assertSame Expected: [[57413], identityHash=1099388258186] Actual: [[57412], identityHash=1099388258194] File search:00:00:13.822000 Suite construction: 00:00:01.95 Test execution: 00:10:05.236000 Total time: 00:10:29.321000 The one failure shows that between the time the record events log was filled and the time of the second asset, 1 record was added to the event log. This is not unusual on a busy system and is perhaps a flaw in the test case, certainly not a bug in the interpreter or the extension.. I don't have a Linux system handy to re-test on right now. But, I didn't see those failures at the time of the release. So - I say build the AIX package with the release code. Open a bug if the test suite continues to fail on the CHAR tests. Just be sure you are using the actual release source: https://oorexx.svn.sourceforge.net/svnroot/oorexx/main/releases/4.1.1/trunk Thanks again for building the AIX packages Rainer -- Mark Miesfeld -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
[Oorexx-devel] Sourceforge
Who is master of ooRexx on Sourceforge? The examples section on it is deplorable Maybe one could now point at rosettacode.org (as Rick prodused tons of examples) Found this address on sourceforge. iWeaveRexx@users.sourceforge Sent a question: Who are you? Walter Back came _ From: Mail Delivery System [mailto:mailer-dae...@mucimg02.unicreditgroup.eu] Sent: Thursday, June 14, 2012 9:18 AM To: PACHL Walter (Ext. - UBIS - UniCredit) Subject: Undeliverable: Rexx Samples Delivery has failed to these recipients or distribution lists: iWeaveRexx@users.sourceforge A problem occurred during the delivery of this message. Microsoft Exchange will not try to redeliver this message for you. Please try resending this message later, or provide the following diagnostic text to your system administrator. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
Re: [Oorexx-devel] ooSQLite usage
Found one strange thing
The program creates an array of SQL insert statements, after parse of the
source file.
This array is then used to do the actuall insert in the memory database (the
array is also saved to a file)
memDB = .ooSQLiteConnection~new(:memory:)
discDB = .ooSQLiteBackup~new(memDB, fileDB)
..
do sql over mycmds
stmt = .ooSQLiteStmt~new(memDB, sql)
stmt~step
stmt~finalize
end
and then
/* here we do the backup to disc */
ret = discdb~step(-1)
if ret == discDB~DONE then do
say 'Back up finished with no error:' --discDB~lastErrCode
discDB~lastErrMsg
end
memDB~close
What's happening is that one empty record is inserted in the database !
creating an in memory database with ooSQlLite3 ( ooSQLite :memory:)
and create table and then loading the table with .read mycmds.txt will not
create an empy row in memory database
the last 2 record of input file (array) is
INSERT INTO LIDS (lid, uid, empid, Name, privileges) VALUES('x',
yyy ', ' z', 'name', privileges'); -- can't show
real values
--empty line -- (just empty x'0D0A' )
file is created with
f =.stream~new(mycmds.txt)
do line over mycmds/* Write the lines */
f~lineout(line)
end
f~close
-
Ursprungligt Meddelande:
Från: hakan hexi...@users.sourceforge.net
Till: oorexx-devel@lists.sourceforge.net
Kopia:
Datum: torsdag, 14 juni 2012 21:22
Ämne: Re: [Oorexx-devel] ooSQLite usage
As I saw that ooSQLite was updated in incubator, regarding the SQLite backup
API,I downloaded the source and compiled it under windows 7 64bit. From the
sample /testing/loadDatabase.rex, I made my own database loading rex.
The program creates a sqlite database in memory, with 6 columns, read and parse
a 181Mb file and creates SQL insert statements, then inserts the resulting
records (101657 records) into the memory database and last step save the in
memory database to disc.
This programs elapsed time was around 61-65 seconds( including copy to disc) ,
compared to the previous ~17 MINUTES when using a disc based database only,
with same data.
Great improvment.
By the way I also downloaded the latest SQLite source 3.7.13 and used that in
the compile above.
All works very well.
/hex
-
Ursprungligt Meddelande:
Från: hakan hexi...@users.sourceforge.net
Till: Mark Miesfeld miesf...@gmail.com
Kopia:
Datum: fredag, 08 juni 2012 22:20
Ämne: Re: [Oorexx-devel] ooSQLite usage
Well, tried with BEGIN TRANSACTION. COMMIT and cut the time by around 5
minutes, so when initial loading is done, it seems that inmemory database is
much quickier, because of hardisc speed, if a possibility to save the inmemory
database to harddisc is available.
Looking forward to your implementation of the .backup (.dot ) commands in
ooSQLite.
Keep up your good work (as a side note I am very satisfied with ooDialog 4.2
excellent work, I love it, maybe as I now understand how it works (at least to
some point) and how to use it together with ooRexx, OO is hard for an old
mainframer :-) )
/hex
-
Ursprungligt Meddelande:
Från: Mark Miesfeld miesf...@gmail.com
Till: hexi...@users.sourceforge.net, Open Object Rexx Developer Mailing List
oorexx-devel@lists.sourceforge.net
Kopia:
Datum: fredag, 08 juni 2012 21:13
Ämne: Re: [Oorexx-devel] ooSQLite usage
Very cool.
On Fri, Jun 8, 2012 at 11:07 AM, hakan hexi...@users.sourceforge.net wrote:
After trying the ooSQLite package, I must say I am very impressed (well I am
not a DB person, so the learning curve is step) but I find it useful for
some of my ooREXX programs.
The samples have been very helpful, I am now able to create, load and query
a database/tables in ooREXX.
One thing is performance, when creating a new database, I have read/parsed a
file and then loaded it (92 000 rows) to a database/table , took around 17
minutes on a i3 2.13Ghz !
From my reading (but I'm also not a DB person) this is a problem with
INSERTS with not using an explicit BEGIN TRANSACTION COMMIT.
As I understand it, for every individual SQL statement executed,
SQLite will implicitly add the BEGIN TRANSACTION / COMMIT if it is not
already within a transaction. The SQLite engine only does about 4 or
5 inserts a second under this condition.
If you add the explicit BEGIN TRANSACTION before the first insert and
then end with COMMIT the speed should be more normal. Something like
this:
BEGIN TRANSACTION;
INSERT INTO episodes VALUES(0, NULL, 'Good News Bad News');
INSERT INTO episodes VALUES(1, 1, 'Male Unbonding');
INSERT INTO episodes VALUES(2, 1, 'The Stake Out');
INSERT INTO episodes VALUES(3, 1, 'The Robbery');
...
INSERT INTO food_types VALUES(13, 'Seafood');
INSERT INTO food_types VALUES(14, 'Soup');
INSERT INTO food_types VALUES(15, 'Vegetables');
COMMIT;
I guess it is COMMIT not END TRANSACTION. I first thought it was END
TRANSACTION.
The above is the raw SQL, I'm assuming
Re: [Oorexx-devel] ooSQLite usage
On Thu, Jun 14, 2012 at 1:11 PM, hakan hexi...@users.sourceforge.net wrote:
Found one strange thing
The program creates an array of SQL insert statements, after parse of the
source file.
This array is then used to do the actuall insert in the memory database (the
array is also saved to a file)
memDB = .ooSQLiteConnection~new(:memory:)
discDB = .ooSQLiteBackup~new(memDB, fileDB)
..
do sql over mycmds
stmt = .ooSQLiteStmt~new(memDB, sql)
stmt~step
stmt~finalize
end
and then
/* here we do the backup to disc */
ret = discdb~step(-1)
if ret == discDB~DONE then do
say 'Back up finished with no error:' --discDB~lastErrCode
discDB~lastErrMsg
end
memDB~close
What's happening is that one empty record is inserted in the database !
creating an in memory database with ooSQlLite3 ( ooSQLite :memory:)
and create table and then loading the table with .read mycmds.txt will not
create an empy row in memory database
the last 2 record of input file (array) is
INSERT INTO LIDS (lid, uid, empid, Name, privileges) VALUES('x',
yyy ', ' z', 'name', privileges'); -- can't
show real values
--empty line -- (just empty x'0D0A' )
Not sure I understand what is happening. Does the empty line get passed into:
stmt = .ooSQLiteStmt~new(memDB, sql)
If you use the same file but load the records the way you first did,
not using an in-memory database, do you still get an empty record?
If you take the same file and chop it down to, say just the last 10
lines in file, do you still get the empty record?
--
Mark Miesfeld
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel
Re: [Oorexx-devel] ooSQLite usage
False alarm,
After rereading my program code I found out that I had a return statement
missing.
After writing the array of commands to a file, there was an item, which was an
empty SQL INSERT statement, added to the array of commands.
Sorry for the confusion.
/hex
-
Ursprungligt Meddelande:
Från: Mark Miesfeld miesf...@gmail.com
Till: hexi...@users.sourceforge.net, Open Object Rexx Developer Mailing List
oorexx-devel@lists.sourceforge.net
Kopia:
Datum: fredag, 15 juni 2012 01:26
Ämne: Re: [Oorexx-devel] ooSQLite usage
On Thu, Jun 14, 2012 at 1:11 PM, hakan hexi...@users.sourceforge.net wrote:
Found one strange thing
The program creates an array of SQL insert statements, after parse of the
source file.
This array is then used to do the actuall insert in the memory database (the
array is also saved to a file)
memDB = .ooSQLiteConnection~new(:memory:)
discDB = .ooSQLiteBackup~new(memDB, fileDB)
..
do sql over mycmds
stmt = .ooSQLiteStmt~new(memDB, sql)
stmt~step
stmt~finalize
end
and then
/* here we do the backup to disc */
ret = discdb~step(-1)
if ret == discDB~DONE then do
say 'Back up finished with no error:' --discDB~lastErrCode
discDB~lastErrMsg
end
memDB~close
What's happening is that one empty record is inserted in the database !
creating an in memory database with ooSQlLite3 ( ooSQLite :memory:)
and create table and then loading the table with .read mycmds.txt will not
create an empy row in memory database
the last 2 record of input file (array) is
INSERT INTO LIDS (lid, uid, empid, Name, privileges) VALUES('x',
yyy ', ' z', 'name', privileges'); -- can't
show real values
--empty line -- (just empty x'0D0A' )
Not sure I understand what is happening. Does the empty line get passed into:
stmt = .ooSQLiteStmt~new(memDB, sql)
If you use the same file but load the records the way you first did,
not using an in-memory database, do you still get an empty record?
If you take the same file and chop it down to, say just the last 10
lines in file, do you still get the empty record?
--
Mark Miesfeld
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel
