Re: [Firebird-devel] Named parameters in client libraries
On 19/08/2022 10:55, Adriano dos Santos Fernandes wrote: On 19/08/2022 06:06, Tony Whyman wrote: On 19/08/2022 00:42, Adriano dos Santos Fernandes wrote: On 18/08/2022 18:53, Tony Whyman wrote: IBX for Lazaus follows the original Deplhi IBX approach of adopting PSQL syntax for named parameters. Can a name be used more than once, and if yes, how is its type deduction? Adriano Uniqueness of parameters names is not required. If you have two or parameters with the same name in a given SQL Statement then both get set at the same time and to the same value. Type conversions are an issue for Firebird rather than IBX and are handled identically to positional parameters. Can't the user's library inspect parameters and its type? If yes, how would it be returned if a parameter name is used more than once with different Firebird types deduced in the different question-marks generated? The user can always check the current SQL Type of a parameter and use the setter they believe is most appropriate. If they insist on giving the same parameter name to two or more different parameters with differing SQL types - then that's their problem. The ByName method only ever returns the first parameter with a given name in position order. If the user calls its setter then the same value is given to all parameters with the same name. If there are no type compatibility problems then it works. If there are type compatibility issues then an exception occurs when they execute the query. The correct solution is to use different parameters names and set each parameter appropriately. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Named parameters in client libraries
On 19/08/2022 00:42, Adriano dos Santos Fernandes wrote:
On 18/08/2022 18:53, Tony Whyman wrote:
IBX for Lazaus follows the original Deplhi IBX approach of adopting PSQL
syntax for named parameters.
Can a name be used more than once, and if yes, how is its type deduction?
Adriano
Uniqueness of parameters names is not required. If you have two or
parameters with the same name in a given SQL Statement then both get set
at the same time and to the same value.
Type conversions are an issue for Firebird rather than IBX and are
handled identically to positional parameters.
The general algorithm for SQL statement execution is:
1. Parse the statement replacing named parameters with '?' placeholders
and create a name to parameter position index.
2. Call Firebird to prepare the query.
3. Set up a parameter array data structure using the Firebird input
parameter metadata, indexed by both position (integer) and name.
4. The user sets parameter values in any order. There are as many setter
methods for a parameter as types available (e.g. AsString, AsInteger,
AsFloat, AsDateTime, etc). The parameter's SQL Type is originally
derived from the Firebird metadata but may be overridden at this point
if the setter used is for a different SQL Type. For string setters, the
Pascal AnsiString type includes the codepage identifier (UTF8 is a
codepage for this purpose), and this is used to set the parameter's
character set id.
5. Immediately prior to executing the query, the input parameter
metadata is regenerated from the current SQL Types in the parameter
array, and the parameter buffer is packed using the regenerated metadata.
6. The query is executed. Any type conversion problems are reported back
by Firebird and result in an exception.
Note that batch queries are handled similarly to the above, except that
from the second row onwards, the same setter method must be used for
each parameter as was used for the first row - otherwise an exception is
raised.
A simple example (note IAttachment, etc. are Pascal interfaces (managed
types) and not the same as the firebird.pas types):
procedure DoQuery(Attachment: IAttachment);
var Transaction: ITransaction;
Statement: IStatement;
begin
Transaction :=
Attachment.StartTransaction([isc_tpb_write,isc_tpb_nowait,isc_tpb_concurrency],taRollback);
Statement := Attachment.PrepareWithNamedParameters(Transaction,'INSERT INTO
EMPLOYEE (EMP_NO, FIRST_NAME, LAST_NAME, PHONE_EXT, HIRE_DATE,' +
'DEPT_NO, JOB_CODE, JOB_GRADE, JOB_COUNTRY, SALARY) '+
'VALUES (:EMP_NO, :FIRST_NAME, :LAST_NAME, :PHONE_EXT, :HIRE_DATE,' +
':DEPT_NO, :JOB_CODE, :JOB_GRADE, :JOB_COUNTRY, :SALARY)');
with Statement.SQLParams do
begin
ByName('EMP_NO').AsInteger := 150;
ByName('LAST_NAME').AsString := 'Doe';
ByName('FIRST_NAME').AsString := 'John';
ByName('PHONE_EXT').AsString := '';
ByName('HIRE_DATE').AsString := '2015-4-1';
ByName('DEPT_NO').AsString := '600';
ByName('JOB_CODE').AsString := 'Eng';
ByName('JOB_GRADE').AsInteger := 4;
ByName('JOB_COUNTRY').AsString := 'England';
ByName('SALARY').AsFloat := 41000.89;
end;
Statement.Execute;
end;
Regards
Tony
Firebird-Devel mailing list, web interface
athttps://lists.sourceforge.net/lists/listinfo/firebird-devel
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Named parameters in client libraries
IBX for Lazaus follows the original Deplhi IBX approach of adopting PSQL syntax for named parameters. For example:Select * From EMPLOYEE where EMP_NO = :KEYVALUE;It also allows for parameter names that are case sensitive and which contain special characters by enclosing the parameter name in double quotes e.g.Select * From EMPLOYEE where EMP_NO = :"Key Value";Named parameter handling takes place in the underlying Firebird Pascal API.Additionally, in IBX itself, the SQL update queries defined for a buffered dataset (TDataset descendents) allow parameter names to be preceded by "OLD_" or "NEW_". When a row in a buffered dataset is being edited, the original row values are saved and can be accessed using the "OLD_" prefix. The current values are returned by default or when "NEW_" is used as a prefix (effectively "NEW_" is not needed but included for completeness). e.g.UPDATE EMPLOYEE Set EMP_NO = :EMP_NO, FIRST_NAME = :FIRST_NAME where EMP_NO = :OLD_EMP_NO;The above updates the FIRST_NAME and the EMP_NO with any changes but always uses the old value of EMP_NO in order to update the correct row,If Firebird did support named parameters in a new version, I am not sure if this would be used by IBX. The code already works well and would have to be retained for the support of older versions of Firebird. The TDataset conventions would also have to continue to be supported.RegardsTony Whyman Original message From: Adriano dos Santos Fernandes Date: 18/08/2022 03:03 (GMT+00:00) To: For discussion among Firebird Developers Subject: [Firebird-devel] Named parameters in client libraries Hi Mark, Jiri, all!How are client libraries (Jaybird, .NET Provider, Delphi ones)describing its named parameters to their users?I mean, given this SQL:select * from rdb$database where :param = 1 or :param = '2'We have here single name used in context with multiple types.I suppose you transform this to:select * from rdb$database where ? = 1 or ? = '2'Which will map to two Firebird parameters with different types.But for the user of the library, I suppose it's one parameter, correct?And what type (and the deduction rules) this parameter will be described as?AdrianoFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-develFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Are isc_sql_interprete and isc_sqlcode deprecated calls?
to be precise - after code review I do not see isc_sqlcode() to be maked as deprecated but I remember well that it was mentioned somewhere as not recommended in new development, sqlstate is preferred The Firebird 4 language reference states: |"SQLCODE| has been used for many years and should be considered as deprecated now. Support for |SQLCODE| is likely to be dropped in a future version." This probably should be reflected in the code. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Are isc_sql_interprete and isc_sqlcode deprecated calls?
Aren't you confusing isc_interprete with isc_sql_interprete? isc_sql_interprete interpets the SQLCode, so it does make sense to drop it if you are getting rid of SQLCode. On 17/01/2022 09:54, Alex Peshkoff via Firebird-devel wrote: On 1/17/22 12:24, Tony Whyman wrote: In the legacy Firebird API, there are three functions that support error message formatting: isc_sqlcode, isc_sql_interprete, and fb_interpret. In the OO API, fb_interpret becomes IUtil->formatStatus*. *However, I can find no OO API equivalent for the other two. Is this because they are deprecated for future use? Yes. Whole sqlcode feature is deprecated, sqlstate should be used instead. What about isc_sql_interprete that's just old, unsafe variant of fb_interpret. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Are isc_sql_interprete and isc_sqlcode deprecated calls?
In the legacy Firebird API, there are three functions that support error message formatting: isc_sqlcode, isc_sql_interprete, and fb_interpret. In the OO API, fb_interpret becomes IUtil->formatStatus*. *However, I can find no OO API equivalent for the other two. Is this because they are deprecated for future use? If not, then shouldn't they also be available via IUtil? Tony Whyman MWA Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird and DSQL Scrollable Cursors
On 08/12/2021 09:13, Dmitry Yemanov wrote: 3) "row count" makes it possible to know the position after fetchLast() and everything else could be calculated locally by the client library, thus making the server-supported "current position" totally unnecessary. Do I miss anything? Could we agree on having only "row count" returned via op_info_cursor and leaving "cursor position" (getRow() in Java API) up the connectivity library developers? As long as row count can be returned this would work for IBX. In all other cases, it looks like it should be possible to compute the current row number client side. It would also be very useful to get rowcount returned for unidirectional cursors if that was readily possible. At present, it is only possible to get an accurate count of the number of rows in a cursor after you have fetched all of them. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 5 and Update...Returning
That code is very legacy and dates from a time when Update..Returning did not exist. Actually, it works the same as the modern version. My point was about what the user expects. If you execute an Update...Returning with the code you pasted in below, with Firebird 4.0.0 it drops through the SQLExecProcedure and the user can access the returned values immediately afterwards. In Firebird 5, the code executes the SQLSelect case. Note that only if FGoToFirstRecordOnExecute is true will the returned values be available immediately. Otherwise, the user has to call "Next" explicitly. That is why the change to Update...Returning can break user code. In order to try and maintain the TIBSQL semantics for Firebird 5, I need to be able to distinguish between a true select query, an Update...Returning that returns a singleton row and one that returns a cursor. I may be able to find an answer by experimenting with the new behaviour, but that will not stop me moaning that the extension of Update..Returning has been implemented without keeping faith with existing code. On 26/11/2021 16:47, Dimitry Sibiryakov wrote: Tony Whyman wrote 26.11.2021 17:13: Legacy code can often appear "insane" with the benefit of hindsight. The problem is that it exists and it is your starting point. You don't break it without good reason. Ok, here is legacy code from TIBSQL.ExecQuery: case FSQLType of SQLSelect: begin Call(FGDSLibrary.isc_dsql_execute2(StatusVector, TRHandle, @FHandle, Database.SQLDialect, FSQLParams.AsXSQLDA, nil), True); Call( FGDSLibrary.isc_dsql_set_cursor_name(StatusVector, @FHandle, PChar(FCursor), 0), True); FOpen := True; FBOF := True; FEOF := False; FRecordCount := 0; if FGoToFirstRecordOnExecute then Next; end; SQLExecProcedure: begin fetch_res := Call(FGDSLibrary.isc_dsql_execute2(StatusVector, TRHandle, @FHandle, Database.SQLDialect, FSQLParams.AsXSQLDA, FSQLRecord.AsXSQLDA), False); . and so on... As you can see it IS sane and won't be broken so if _current_ IBX code differs from it - the one who "improved" it must be blamed. Firebird team cannot help in this case. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 5 and Update...Returning
On 26/11/2021 15:43, Dimitry Sibiryakov wrote: Tony Whyman wrote 26.11.2021 16:32: What you appear to be saying is that you have changed/expanded the semantic of Update...Returning, changed the SQL Statement type returned and then not expected the change to break any existing code... Correction: any sane code. There is an old joke that starts with one person asking another the way to Edinburgh. The answer given is "I wouldn't start from here if I were you". That answer may be true, but very unhelpful. Legacy code can often appear "insane" with the benefit of hindsight. The problem is that it exists and it is your starting point. You don't break it without good reason. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 5 and Update...Returning
What you appear to be saying is that you have changed/expanded the semantic of Update...Returning, changed the SQL Statement type returned and then not expected the change to break any existing code... On 26/11/2021 15:01, Dimitry Sibiryakov wrote: Tony Whyman wrote 26.11.2021 15:28: 3. Prior to calling IStatement.Execute, the statement type is checked. The current (IBX) code raises an exception if the sql statement type is isc_info_sql_stmt_select in order to stop the wrong IStatement method being called. So it is actually the test suite that makes (now wrong) assumption that UPDATE...RETURNING has to be described as isc_info_sql_stmt_exec_procedure is wrong. But it used to be the correct assumption. 3. IBX tries to be general purpose and so prepares a statement, then checks the statement type, and then calls the appropriate IStatement method. Changing the statement type semantics broke the logic used to determine which method to call. That's why the change. IBX should check the statement type (which now is isc_info_sql_stmt_select) and call appropriate method i.e. openCursor(). What's wrong? The problem is in TIBSQL. This is a class that goes all the way back to Borland days. It has a common method "ExecQuery" that is called regardless of the statement type. If it is used to open a cursor, then the user either has to set the GoToFirstRecordOnExecute property to ensure that the cursor is automatically positioned on the first row, or explicitly call "Next". If a singleton row is expected then there is no need to do either of the above. Right now, I am not sure how this behaviour can be maintained (and not get inundated by users complaining that their code is broken) without being able to tell the difference between an Update...Returning that returns a singleton row from one that returns a cursor. 4. With reference to item 1 above, if "Update..Returning" returns a statement type of isc_info_sql_stmt_select then you _do_ have to also parse the statement in IBX so that you know its an UPDATE and IStatement.execute has to be called instead of IStatement.openCursor. No, openCursor() must be called here, not execute(). Which is, of course, new behaviour. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 5 and Update...Returning
On 26/11/2021 14:32, Dmitry Yemanov wrote: 26.11.2021 17:28, Tony Whyman wrote: 1. IAttachment.prepare is used to parse 'Update Employee Set Hire_Date = ? Where EMP_NO = ? Returning LAST_NAME' 2. IStatement.getType is then used to determine the statement type. OK so far. 3. Prior to calling IStatement.Execute, the statement type is checked. The current (IBX) code raises an exception if the sql statement type is isc_info_sql_stmt_select in order to stop the wrong IStatement method being called. Why? How is it different from SELECT returning isc_info_sql_stmt_select? They should work the same way. Is the SQL text also parsed to detect UPDATE and that conflicts with the returned statement type? Dmitry 1. This is not a backwards compatible change. If "Update...Returning" had always been given a statement type of isc_info_sql_stmt_select then the IBX code would have been written to support this. As it happens, it was not and so changing the statement type immediately introduces a backwards compatibility issue. Do you have a good reason to make this change? 2. IStatement has different methods "execute" and "openCursor". You need to know in advance which one to call. The former is called when you expect a singleton row (or no output) and the latter when you expect a cursor. 3. IBX tries to be general purpose and so prepares a statement, then checks the statement type, and then calls the appropriate IStatement method. Changing the statement type semantics broke the logic used to determine which method to call. 4. With reference to item 1 above, if "Update..Returning" returns a statement type of isc_info_sql_stmt_select then you _do_ have to also parse the statement in IBX so that you know its an UPDATE and IStatement.execute has to be called instead of IStatement.openCursor. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 5 and Update...Returning
On 26/11/2021 14:16, Dimitry Sibiryakov wrote: Tony Whyman wrote 26.11.2021 15:10: This caused an error message to be generated because only a singleton row was expected and not a cursor. Well, all I can say is that "existing connectivity drivers do not support this feature automagically". Is it really then intended behaviour for Update...Returning to now return an SQL Type of isc_info_sql_stmt_select? Yes. What exactly call produces the error message and what exactly is the error? 1. IAttachment.prepare is used to parse 'Update Employee Set Hire_Date = ? Where EMP_NO = ? Returning LAST_NAME' 2. IStatement.getType is then used to determine the statement type. 3. Prior to calling IStatement.Execute, the statement type is checked. The current (IBX) code raises an exception if the sql statement type is isc_info_sql_stmt_select in order to stop the wrong IStatement method being called. Note that the exception is called by IBX and not Firebird. However, the exception is consequential on Firebird returning the statement type isc_info_sql_stmt_select when all previous versions of Firebird had returned isc_info_sql_stmt_exec_procedure for an Update...Returning statement. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Firebird 5 and Update...Returning
When testing out Dimtry's recent fix for scrollable cursors (adding support for the remote protocol), I ran the full IBX test suite on the current "master" branch. This flagged up a backwards compatibility issue with "Update...Returning". When the statement: 'Update Employee Set Hire_Date = ? Where EMP_NO = ? Returning LAST_NAME' was prepared, the SQL type was returned as isc_info_sql_stmt_select. The previous behaviour was to return isc_info_sql_stmt_exec_procedure. This caused an error message to be generated because only a singleton row was expected and not a cursor. The file doc/sql.extensions/README.returning has been updated and a quick diff with 4.0.0 returned, amongst other changes: 45,46c45,47 < isc_info_sql_stmt_exec_procedure by the API (instead of isc_info_sql_stmt_insert), < so the existing connectivity drivers should support this feature automagically. --- > isc_info_sql_stmt_exec_procedure by the API (for INSERT INTO ... VALUES and statements > with WHERE CURRENT OF) and isc_info_sql_stmt_select for the others statements, so the > existing connectivity drivers should support this feature automagically. Well, all I can say is that "existing connectivity drivers do not support this feature automagically". Is it really then intended behaviour for Update...Returning to now return an SQL Type of isc_info_sql_stmt_select? Tony Whyman MWA Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird and DSQL Scrollable Cursors
Dimtry, Just compiled the updated master branch and tested with IBX. All looks good with the test suite returning the same set of results for bot remote and local databases. Many thanks for the good work. I've noticed that the ODS is now 13.1 compared with 13.0 for Firebird 4.0.0. Is this due to your patch or have there been other changes that have needed this? Regards Tony Whyman On 26/11/2021 09:10, Dmitry Yemanov wrote: Mark et al, Yes, I think that is perfectly acceptable for an initial version. Scrollable cursors are a bit of an oddity anyway, but having scrollable cursors in embedded access, but not in remote access is IMHO less acceptable than bad performance. As long as the performance behaviour is clearly documented, people can make the decision if the bad performance is worth the utility of scrollable cursors for themselves. I've just committed network support for scrollable cursors to master. Prefetch logic is supported for NEXT/PRIOR navigations. Protocol changes are documented here: https://github.com/FirebirdSQL/firebird/issues/7051 The test set to cover different navigation scenarios is being created, once it's ready we'll validate the results between PSQL cursors, embedded access, network access w/prefetch and network access wo/prefetch (FOR UPDATE added to SELECTs). Obviously, we expect them to match ;-) I'd appreciate if Mark could test scrollability from the Jaybird side too (i.e. without fbclient involved), but the new protocol should be supported for that. The committed version is linked to protocol 17 which was introduced for 4.0.1. Thus technically, this improvement can be backported and released together with v4.0.1. However, I'm not sure this risk is acceptable for a point release and I'm ready to introduce protocol 18 in master for v5. Opinions, please. Dmitry Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird and DSQL Scrollable Cursors
Thanks guys for responding. The only issue I have is that the release notes don't seem to tell you that DSQL scrollable cursors are only available for local databases. On 23/10/2021 14:14, Dimitry Sibiryakov wrote: Mark Rotteveel wrote 23.10.2021 14:31: I'm not sure why not. Because of record buffering it is too hard for a feature with so little usage. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Firebird and DSQL Scrollable Cursors
I have been testing out the scrollable cursors API and working with the example employee database. What I am seeing is that scrollable cursors only seem to work with local databases and not remote databases. Is this true? There seems to be nothing in the release notes to suggest this. I am doing a simple test, using 'Select * from EMPLOYEE order by EMP_NO' as the query string and calling OpenCursor with CURSOR_TYPE_SCROLLABLE. If I open the database with the connect string "employee" then FetchPrior etc all seem to work OK. If, instead, I use the connect string inet://localhost/employee, then I get a "Feature not supported" error message with calling FetchPrior (FetchNExt still works). I see the same behaviour with Firebird 3.0.5 and 4.0.0. Tony Whyman MWA Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] The IBatch Interface and Buffer Overflow
On 23/08/2021 13:03, Alex Peshkoff via Firebird-devel wrote: The minor bug is that if TAG_BUFFER_BYTES_SIZE is set to > 256MB no error status is reported on a call to createBatch. Instead, you only know you have a problem when an error is returned from IBatch::add - looks like the buffer size is simply limited to 256MB. Ahh, that's pretty easy to fix but that's traditional approach of using constant as 'hint' and continue operation. For example: SQL> CREATE DATABASE 'qq' pagesize 10; SQL> show db; Database: qq Owner: SYSDBA PAGE_SIZE 32768 .. Same for conf files - invalid values are treated as defaults. So chaging behavior in single place - not sure it's good idea. True, but the difference here is that while I can check the actual page size, I can't check the batch buffer size. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] The IBatch Interface and Buffer Overflow
The good news is that I believe I have found a way around the problem. The bad news is that I think I have found another minor bug. The workaround for variable message sizes is to compute the "used" buffer size after each call to IBatch::add by ensuring that getAlignedLength() is refreshed for each message buffer and adding this to the running total. It is then possible to identify when the buffer will overflow before IBatch::add is called and take appropriate action. The minor bug is that if TAG_BUFFER_BYTES_SIZE is set to > 256MB no error status is reported on a call to createBatch. Instead, you only know you have a problem when an error is returned from IBatch::add - looks like the buffer size is simply limited to 256MB. On 21/08/2021 12:35, Tony Whyman wrote: On 20/08/2021 16:18, Alex Peshkoff via Firebird-devel wrote: On 8/20/21 4:01 PM, Tony Whyman wrote: You can specify such limit - use IBatch::TAG_BUFFER_BYTES_SIZE parameter in batch parameters block when creating a batch, default is 16Mb, hard limit - 256Mb. Ywo such buffers will be available - one for messages data, other - for blobs. Well, in theory OOM can take place later, but that is abnormal case. Alternatively, I could also work with an IBatch method that told me the minimum number of guaranteed calls to IBatch::add (for a given message buffer size). Other suggestions are welcome. Divide what you've requested in TAG_BUFFER_BYTES_SIZE by getAlignedLength() of your message. But be aware that this will not work when/if we move to variable message size and string field with unrestricted length. And this is where the problem lies. We need a long term strategy. One possible strategy would be to choose a number (e.g. 100), multiply it by getAlignedLength() and then set TAG_BUFFER_BYTES_SIZE to either the assumed default (16MB) or a larger value if 100* getAlignedLength() > 16MB. The problem is that, as you note, this will fail "when/if we move to variable message size". Perhaps a getMaxAlignedLength() is needed, or a TAG to force fixed alignment? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] The IBatch Interface and Buffer Overflow
On 20/08/2021 16:18, Alex Peshkoff via Firebird-devel wrote: On 8/20/21 4:01 PM, Tony Whyman wrote: You can specify such limit - use IBatch::TAG_BUFFER_BYTES_SIZE parameter in batch parameters block when creating a batch, default is 16Mb, hard limit - 256Mb. Ywo such buffers will be available - one for messages data, other - for blobs. Well, in theory OOM can take place later, but that is abnormal case. Alternatively, I could also work with an IBatch method that told me the minimum number of guaranteed calls to IBatch::add (for a given message buffer size). Other suggestions are welcome. Divide what you've requested in TAG_BUFFER_BYTES_SIZE by getAlignedLength() of your message. But be aware that this will not work when/if we move to variable message size and string field with unrestricted length. And this is where the problem lies. We need a long term strategy. One possible strategy would be to choose a number (e.g. 100), multiply it by getAlignedLength() and then set TAG_BUFFER_BYTES_SIZE to either the assumed default (16MB) or a larger value if 100* getAlignedLength() > 16MB. The problem is that, as you note, this will fail "when/if we move to variable message size". Perhaps a getMaxAlignedLength() is needed, or a TAG to force fixed alignment? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] The IBatch Interface and Buffer Overflow
After adding support for the IBatch interface to IBX and then stress testing the implementation, I found that IBatch, when used with a remote server was silently losing data when the buffer size was exceeded. That issue has now been fixed. See - https://github.com/FirebirdSQL/firebird/issues/6900 However, I believe that more needs to be done to resolve the issue and any resolution needs to have a wider discussion. The current situation is IBatch::add returns an error when the internal buffer overflows: * when used with an embedded server, this appears always to happen when IBatch::add is called for the message buffer that causes the overflow. It is thus possible to recover by calling IBatch::execute to apply all the previous message buffers and then starting a new batch and call IBatch::add again for the same message buffer. * when used with a remote server, my understanding is that the error is generated on the server side and is always many messages behind. IBatch::execute can still be called to apply the batch, but only by looking at the completion data can you work out at which point the overflow occurred, and hence from which message buffer you need to restart. My problem is with the remote server case and that it seems to be difficult to determine, in advance, how many message buffers to keep a copy of in order to recover from the error. With my test data, the buffer overflow occurs after several thousand calls to IBatch::add. The number of message buffers lost is of the order of hundreds. I don't want to have to keep a copy of every message buffer as that could result in other out of memory issues. My problem is that the number of message buffers that may be lost on buffer overflow appears to be indeterminate. It would be more useful if when IBatch is created, I could specify a message buffer limit and the internal buffers were allocated in advance. Alternatively, I could also work with an IBatch method that told me the minimum number of guaranteed calls to IBatch::add (for a given message buffer size). Other suggestions are welcome. I believe that a solution is necessary because you cannot use IBatch for any large scale use without knowing what you need to do to recover from a buffer overflow error without losing user data. Tony Whyman MWA Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] The IBatch Interface and inline blobs
So I guess that as long as the string is < 32K and you are not using a segmented blob then it is OK to use SQL_VARYING and not the inline blob facility. On the subject of limits, IBatch does seem to have a silent limit that I am still exploring. I was comparing the time taken to insert 10 rows using single inserts and the Batch interface. Using the Batch interface, only 4061 records were written the table, even though 10 were added (IBatch->add). The number 4061 was confirmed from both a read back after commit and by checking the IBatchCompletionState which reported both processed and updated 4061. 4061 seems an arbitrary number. My original test table was declared as Create Table LotsOfData ( RowID integer not null, theDate TimeStamp, MyText VarChar(1024), Primary Key (RowID) ); and on changing this to Create Table LotsOfData ( RowID integer not null, theDate TimeStamp, MyText VarChar(512), Primary Key (RowID) ); I was able to successfully write 8083 rows. I guess that there is some memory limit that is being hit, and the max mumber of rows that can be added depends on the size of each buffer added to the batch. The problem I have is that this is a silent failure. I am checking the status vector returned by each IBatch->add, and no problem appears to be reported. Should I report this as a bug? On 16/07/2021 13:50, Alex Peshkoff via Firebird-devel wrote: On 7/16/21 12:39 PM, Tony Whyman wrote: I have recently added support for the IBatch interface to IBX and the good news that it all appears to work fine. However, I am still puzzled by why inline blobs exist. For as long as I can remember, Firebird has allowed you to over-ride the input metadata SQLType to any other type that can be converted to the actual column type. For example, SQL_BLOB to SQL_VARYING. You can then pass a relatively short blob string as an SQL_VARYING (character id to OCTETS for binary, and otherwise as appropriate) and with no need to go to all the hard work of actually creating a blob,etc. I assume that it is on the server side that the string gets written into blob storage and would not expect it to be otherwise. In all my testing, passing blob text as SQL_VARYING works fine for both normal update/insert operations and for batches using IBatch. So I am puzzled as to why the IBatch inline blob exists. What am I missing? In some field one can have typically small blobs, but some of them may be bigger than maximum string size (btw, 32K is also not too big object currently). If you use segmented blobs it's also problematic to send them to server in appropriate form using strings. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] The IBatch Interface and inline blobs
I have recently added support for the IBatch interface to IBX and the good news that it all appears to work fine. However, I am still puzzled by why inline blobs exist. For as long as I can remember, Firebird has allowed you to over-ride the input metadata SQLType to any other type that can be converted to the actual column type. For example, SQL_BLOB to SQL_VARYING. You can then pass a relatively short blob string as an SQL_VARYING (character id to OCTETS for binary, and otherwise as appropriate) and with no need to go to all the hard work of actually creating a blob,etc. I assume that it is on the server side that the string gets written into blob storage and would not expect it to be otherwise. In all my testing, passing blob text as SQL_VARYING works fine for both normal update/insert operations and for batches using IBatch. So I am puzzled as to why the IBatch inline blob exists. What am I missing? Tony Whyman MWA Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Interface versions
Hi! We previously decided that we does not need to maintain API compatibility within mistakes done in alpha/beta versions. So interfaces does not need to be versioned between unstable releases. Well, you may not need to maintain API compatibility, but there does need to be a clear and systematic way of signalling which version of an interface you are working with. The IUtil interface is a case in point. For Firebird 3 it's version 13. For Firebird 4 beta 1 it's version 21 and the following got appended to the interface IUtilImpl_vTable.getDecFloat16 := @IUtilImpl_getDecFloat16Dispatcher; IUtilImpl_vTable.getDecFloat34 := @IUtilImpl_getDecFloat34Dispatcher; IUtilImpl_vTable.getTransactionByHandle := @IUtilImpl_getTransactionByHandleDispatcher; IUtilImpl_vTable.getStatementByHandle := @IUtilImpl_getStatementByHandleDispatcher; IUtilImpl_vTable.decodeTimeTz := @IUtilImpl_decodeTimeTzDispatcher; IUtilImpl_vTable.decodeTimeStampTz := @IUtilImpl_decodeTimeStampTzDispatcher; IUtilImpl_vTable.encodeTimeTz := @IUtilImpl_encodeTimeTzDispatcher; IUtilImpl_vTable.encodeTimeStampTz := @IUtilImpl_encodeTimeStampTzDispatcher; For Firebird 4 beta 2, it's version 24 and the interface gets more appended to it i.e. function getInt128(status: IStatus): IInt128; procedure decodeTimeTzEx(status: IStatus; timeTz: ISC_TIME_TZ_EXPtr; hours: CardinalPtr; minutes: CardinalPtr; seconds: CardinalPtr; fractions: CardinalPtr; timeZoneBufferLength: Cardinal; timeZoneBuffer: PAnsiChar); procedure decodeTimeStampTzEx(status: IStatus; timeStampTz: ISC_TIMESTAMP_TZ_EXPtr; year: CardinalPtr; month: CardinalPtr; day: CardinalPtr; hours: CardinalPtr; minutes: CardinalPtr; seconds: CardinalPtr; fractions: CardinalPtr; timeZoneBufferLength: Cardinal; timeZoneBuffer: PAnsiChar); For Firebird 4 RC1, the version number gets reset to 4 and the Firebird 4 extensions change such that IUtilImpl_vTable.getTransactionByHandle := @IUtilImpl_getTransactionByHandleDispatcher; IUtilImpl_vTable.getStatementByHandle := @IUtilImpl_getStatementByHandleDispatcher; are removed. I fell into the usual trap this morning of testing out IBX with an updated Firebird.pas and forgetting to replace the Firebird beta 1 client library. Result a call to what should have been decodeTimeTz came back with a "Transaction Handle error"! This took a little while to work out. The result is that you cannot treat the version number as anything other than an identifier with version 13 and version 4 being currently the only valid values - and a consequential need to update the list of valid values everytime a new version of the Firebird API comes out, rather than being able to rely on a monotonically increasing sequence for backwards compatibility. Perhaps what is really needed is a separate version no and "stable" flag for each interface. On 17/02/2021 15:33, Adriano dos Santos Fernandes wrote: So it does not make sense to me to have more than one "version:" between official releases. We currently have in Util: version: // 3.0 => 4.0 Alpha1 version: // 4.0 Beta1 => 4.0 Beta2 Should we use only one version here for v4? Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Usage of 32b version of Firebird
There is an important point sitting here for support of legacy applications. While a standalone x86 Firebird Server is likely to be of increasingly little interest, an x86 client library may need to be supported for much longer. This then gives rise to the question: what about the embedded server. Does that have to be available for x86? On 02/02/2021 09:25, marius adrian popa wrote: Current Ubuntu LTS is 64 bit , only with selected packages for 32 bit apps (Thanks to Steam and Wine apps) https://www.omgubuntu.co.uk/2019/06/ubuntu-is-dropping-all-32-bit-support-going-forward Even FreeBSD moved i386 to Tier 2 for FreeBSD 13.x https://lists.freebsd.org/pipermail/freebsd-announce/2021-January/002006.html So in the end my guess 32 bit is only for legacy apps (Think Delphi apps without source code ... ) Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Jaybird Srp authentication issue
On 04/12/2020 16:03, Dimitry Sibiryakov wrote: 04.12.2020 16:20, Mark Rotteveel wrote: After closer inspection, I found the issue. The SHA-1 hash of DAVIDS is 00AD377F8297F04FD83DFDBF48AABF316850862F. Seeing that leading zero, I guessed that might be part of the problem. After stripping the leading zero from the user hash in Jaybird, the authentication succeeds. The roundtrip from hash bytes to BigInteger back to bytes as hash input (in RemotePassword::clientProof (srp.cpp) and makeProof (srp.h)), probably strips any leading zero byte(s). So the question now is whether it is a bug in Firebird Srp implementation or Jaybird one. I would say the former. Does the same problem exist with Firebird SRP and SHA-256. This uses a different codebase to SHA-1, so it would be interesting to know whether the problem is specific to SHA-1. It also begs the question: if you are serious about SRP security then why are you still using SHA-1? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Jaybird Srp authentication issue
On 04/12/2020 14:55, Adriano dos Santos Fernandes wrote: On 04/12/2020 11:48, Mark Rotteveel wrote: This behaviour is a security issue, it leaks existence or non-existence of the user. Is it a security issue in any website that if I try to create an account and it says the user already exist? It is if you are not logged in already! Account creation should only be possible for a logged in user and only if they have sufficient privilege. Ideally, an account creation attempt should also be logged. However, if I recall, this is an area that needs work in Firebird - see CORE-5786. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Jaybird Srp authentication issue
Is this a character case problem? A quick look at the Firebird SRP code suggests that the username/password hash is generated using the actual character string without any transformation to all upper case. On 04/12/2020 13:38, Mark Rotteveel wrote: A bug was reported today against Jaybird (http://tracker.firebirdsql.org/browse/JDBC-635), that certain Srp users cannot authenticate against Firebird 3 when using Jaybird. The issue is not reproducible with fbclient. This obviously means that there is something wrong in Jaybird's SRP implementation, but so far I have no clue as to what. The reported problem occurs with the user name DAVIDS (and one other username that wasn't provided). I can report the problem locally with this username. Any ideas, or tips how to debug this? Mark Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] OO API examples for Object Pascal
sent. It is now used to determine if the Firebird 3 OO API is present and in the latest (beta) versions to adapt to the use of the Firebird 4 extensions. It also allows the same system to host multiple copies of the Firebird library - I exploit this to readily permit automated testing against different versions of the Firebird Client library. I guess examples will only scratch the surface of these issues. You will probably end up concentrating on opening an example database using a basic DPB and reading/writing a limited set of data types. Regards Tony Whyman MWA On 17/08/2020 16:25, Paul Reeves wrote: I thought I would port the OO API examples to Object Pascal. After all, they are quite simple, so how hard could it be? In fact it is more difficult than I thought and I am not happy with the results. I've done two examples - update and select - and I would like to commit them but before doing so I'd like to discuss how we should proceed. In my opinion we should start a new sub-directory to contain all the object pascal code. This is because the examples will actually need several supporting files in a common directory. It would be best if the were kept clearly separate from other examples. But the main problem I have found is that if I try to copy the C++ style too closely all we end up with is very bad object pascal. It succeeds in demonstrating the new API but does not demonstrate good programming practice. However, if we want to demonstrate good programming practice we should probably integrate the examples into the VCL. And that might be a step too far. Does anyone have any thoughts on this? Paul Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Crazy about time/timestamp with timezone
On 08/07/2020 14:00, Pavel Cisar wrote: TIME WITH TIMEZONE is pointless. It sort off works for "naive" TZ that does not have things like summer time and other time shifts based on date. The pytz library does not even allow you to create time with such tz. The dateutil does, but the usability is near zero, because you can't get offset, tz name etc from it (so also the calculations are crewed). I have also been struggling to find a use for TIME WITH TIME ZONE. For example, if you are dealing with "Wall clock time" (e.g. the opening time of a shop) then you really want to have the time zone encoded separately. This is because, if you want know (e.g.) in London, if a shop in (e.g.) New York is open, you need to translate both your local time and the shop local time to GMT on the day you are making the query and not on some arbitrary date, such as when the opening time was entered into the database or 2020-01-01. In such an example, you really do not want to convert to GMT on data input - you wait until the query is performed. I can find examples of TIME WITH TIME ZONE when the date for a conversion to GMT is specified at data input (e.g. when inputting the time code on a log file - as a shortform timestamp), but not for conversion on some fixed date. I would be happy to drop TIME WITH TIME ZONE. On the other hand, my example would benefit from a built-in function to convert a local time (in the database) and its time zone to GMT on a specific date (e.g. today). Rather than encode time zone names in the database - when the time zone is given as a separate column, I would specify a Domain for the Firebird Time Zone ID in the RDB$TIME_ZONES file and use that for the column value. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB 4, iUtil and new interface in general
On 15/06/2020 15:37, Alex Peshkoff via Firebird-devel wrote:
On 2020-06-11 18:30, Tony Whyman wrote:
On 11/06/2020 11:18, Pavel Cisar wrote:
This is not a problem for languages like Python, Java and .NET that
don't use these files at all, but C++ & Pascal are still important
languages and we should offer some better solution. At least it
should be discussed with those who use them (for example MWA
Software, developer of ibx4lazarus?).
Did I just hear my name mentioned ;)
My main gripe with the new API is really more about documentation
than anything else. It is not easy to work out how to use it except
by experimentation. E.g. with the new DecFloat type I basically had
to look at how the interface returned the BCD format value by using
the debugger to inspect the data at the byte level and then work out
how to convert this (unpacked) format into the packed tBCD record
used by FPC and Delphi - and vice versa.
That's format present in original IBM's library. They do not support
packed BCD. If you provide a link to the description of packed BCD I
can add support for it.
I was more concerned about documenting (or providing a reference to the
documentation) for the BCD format used by the Firebird Client API. When
it comes to the packed BCD format used by Delphi and FPC, there seems to
be little more than the source code in the FmtBCD unit - and they are
not even consistent in the way that the precision is recorded.
I still don't know how to process an INT128 type except by converting
it to a string (!) and then parsing the string.
It's compatible with __int128 from gcc. Sorry, I do not know how to
explain it in pascal way.
Well, that gives me a pointer.
The "firebird.pas" file is always "processed" before it is taken into
IBX for Lazarus - which is perhaps why I am less bothered about any
imperfections in the source code. The most important need for this is
to extract the type and const declarations and separate them out in
order to avoid name space conflicts between the actual API and IBX,
as well as to limit pollution of the global name space. I also remove
the exceptions generated by the cloop code as I prefer to handle the
exceptional conditions in IBX itself.
An ability to process exception in another way is very important but
missing. Can you suggest a generic way? If yes suppose we can add it
to code generator.
Here's a simple example of what I have done:
function IMaster.getMetadataBuilder(status: IStatus; fieldCount:
Cardinal): IMetadataBuilder;
begin
Result := MasterVTable(vTable).getMetadataBuilder(Self, status,
fieldCount);
{$IFDEF USEFBEXCEPTION}FbException.checkException(status);{$ENDIF}
end;
I have made every call to FbException.checkException conditional - in
practice, IBX never defines USEFBEXCEPTION and hence these calls are
commented out. The IBX code will typically check the status itself and
raise its own exception when necessary. There are some exceptions e.g.
when creating an attachment, IBX looks at the error code and may perform
an alternative e.g. if the database does not exist then it may be created.
Is it really necessary to perform exception handling in firebird.pas?
The user can either do their own exception handling or call
FbException.checkException to perform default exception handling.
The call to get the IMaster interface also assumes static linking
with the client library,
???
Why?
The syntax
function fb_get_master_interface : IMaster; cdecl; external 'fbclient';
is usually used for static linking. I believe that it can be used with
dlls - but you have no control over the location. You just get the first
one it finds in the path.
while IBX always dynamically loads the fb client library and then
dynamically links to each exported call using GetProcAddress.
fb_get_master_interface() is absolutely same call as all others in
fbclient, I do not understabd your problems with use of it in
GetProcAddress. At least it's posix analogue dlsym() works fine.
I am really just making the point that the call to
fb_get_master_interface in firebird.pas is really little more than a
simple example of how to load the firebird client library. For anything
more than basic use this has to be commented out as it will conflict
with dynamic library loading.
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB 4, iUtil and new interface in general
On 11/06/2020 11:18, Pavel Cisar wrote: This is not a problem for languages like Python, Java and .NET that don't use these files at all, but C++ & Pascal are still important languages and we should offer some better solution. At least it should be discussed with those who use them (for example MWA Software, developer of ibx4lazarus?). Did I just hear my name mentioned ;) My main gripe with the new API is really more about documentation than anything else. It is not easy to work out how to use it except by experimentation. E.g. with the new DecFloat type I basically had to look at how the interface returned the BCD format value by using the debugger to inspect the data at the byte level and then work out how to convert this (unpacked) format into the packed tBCD record used by FPC and Delphi - and vice versa. I still don't know how to process an INT128 type except by converting it to a string (!) and then parsing the string. The "firebird.pas" file is always "processed" before it is taken into IBX for Lazarus - which is perhaps why I am less bothered about any imperfections in the source code. The most important need for this is to extract the type and const declarations and separate them out in order to avoid name space conflicts between the actual API and IBX, as well as to limit pollution of the global name space. I also remove the exceptions generated by the cloop code as I prefer to handle the exceptional conditions in IBX itself. The call to get the IMaster interface also assumes static linking with the client library, while IBX always dynamically loads the fb client library and then dynamically links to each exported call using GetProcAddress. This allows IBX to work on systems with multiple copies of the fb client (very useful for testing) and for the legacy interface this behaviour allowed it to adjust to different versions of InterBase (e.g. before and after the services interface was introduced). The additions to the new API for FB4 don't seem to have introduced any backwards compatibility issues. The same IBX code works with both Firebird 3 and Firebird 4 client libraries and uses each interface's version number to determine whether the FB4 functions should exist. It will not attempt to call an FB4 function if the version number is too low. I do agree that there should be a clear scheme for version numbering as consistent version numbers are really important when it comes to backwards compatibility. Tony Whyman MWA Software Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6328) FB4 Beta 2 may still be using the current date for TIME WITH TIME ZONE and extended wire protocol.
FB4 Beta 2 may still be using the current date for TIME WITH TIME ZONE and extended wire protocol. -- Key: CORE-6328 URL: http://tracker.firebirdsql.org/browse/CORE-6328 Project: Firebird Core Issue Type: Bug Components: API / Client Library Affects Versions: 4.0 Beta 2 Environment: Linux Mint 19.3, Firebird Beta 2 Firebird/Linux/AMD/Intel/x64 (access method), version "LI-T4.0.0.2006 Firebird 4.0 Beta 2" Reporter: Tony Whyman For testing, I have first set up a simple script i.e. Create Database 'localhost:/tmp/test.fdb' USER 'SYSDBA' PASSWORD 'masterkey'; SET BIND OF TIME ZONE TO EXTENDED; Create Table FB4TestData_TZ ( RowID Integer not null, TimeCol TIME WITH TIME ZONE, Primary Key(RowID) ); Insert into FB4TestData_TZ(RowID,TimeCol) Values(1,'11:31:05.0001 America/New_York'); commit; Select * From FB4TestData_TZ; Drop Database; Run using isql, this gives the expected output i.e. '11:31:05.0001 America/New_York' I then investigated the same using the development version of the IBX Firebird Pascal API, focusing on decoding the value of the "TimeCol" column when the extended wire protocol is in use. The Pascal code for decoding the buffer element looks like: procedure DecodeTimeTZEx(bufptr: PByte; OnDate: TDateTime; var time: TDateTime; var dstOffset: smallint; var aTimezone: TFBTimeZoneID; var aTimezone: AnsiString); const bufLength = 128; var Hr, Mt, S, DMs: cardinal; tzBuffer: array[ 0.. bufLength] of AnsiChar; begin UtilIntf.decodeTimeTzEx(StatusIntf,PISC_TIME_TZ_EX(bufptr), @Hr, @Mt, @S, @DMs,bufLength,@tzBuffer); time := FBEncodeTime(Hr, Mt, S, DMs); dstOffset := PISC_TIME_TZ_EX(bufptr)^.ext_offset; timezoneID := PISC_TIME_TZ_EX(bufptr)^.time_zone; aTimezone := strpas(PAnsiChar(@tzBuffer)); end; When I run it through on the debugger, I can see that the buffer is correctly decoded by UtilIntf.decodeTimeTzEx (Hr = 11, etc). However, "dstOffset" is set to -240 and not -300 which would be expected for the time zone at 2020/1/1. If I add to the code: UtilIntf.decodeTime(PISC_TIME_TZ_EX(bufptr)^.utc_time, @Hr, @Mt, @S, @DMs); in order to directly decode the time component in GMT, this is also as expected (i.e. Hr = 16, etc.) So, how did decodeTimeTzEx correctly decode the buffer to the correct local time when the offset provided by the server was out by -60? I then kill the server and change the system date to '2020/1/1' (at the time the test was first run the system date is 2020/6/9 i.e. when daylight savings time is in effect in America/New_York). I now see that the buffer is also correctly decoded by UtilIntf.decodeTimeTzEx (Hr = 11, etc). However, the dstOffset is also now correctly set to -300. i.e. the dstOffset appears to be following daylight savings time rules for the current date and not for a fixed date of 2020/1/1. The only explanation I have is that both server and client were working with the current date and that additionally the client side is ignoring the offset provided in the extended wire protocol. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6326) Date used to convert TIME WITH TIME ZONE to GMT should be configurable
Date used to convert TIME WITH TIME ZONE to GMT should be configurable -- Key: CORE-6326 URL: http://tracker.firebirdsql.org/browse/CORE-6326 Project: Firebird Core Issue Type: Improvement Components: Engine Affects Versions: 4.0 Beta 2 Environment: All Reporter: Tony Whyman A TIME WITH TIME ZONE data type records the time in GMT regardless of whether or not GMT or a local time is used to input the time value. In order to translate a local time to GMT and back again, a date must be assumed so that the daylight savings time adjustment, if any, can be computed. In FB4 Beta 1, this was the CURRENT_DATE. In FB4 Beta 2, this has changed to 2020/1/1. Neither approach is perfect. This is particularly true when the time given is in the context of a specific date which is neither the current date, nor some arbitrary default. It is proposed that while 2020/1/1 is as good a default as any, the date used to translate to and from GMT should be configurable i.e. 1. The server wide default date should be a configurable parameter in the firebird.conf file. 2. An SQL statement (e.g. SET TIME WITH TIME ZONE DATE TO ''; ) should be provided to allow the user to change the default in the context of the current connection and/or transaction. This statement will need to be recognised and actioned by both the server (e.g. for time with time zone values given as a literal) and the client (for translating input and output time with time zone parameter values). 3.The date used to translate to and from GMT should be available as a column in the MON$ATTACHMENTS and the MON$TRANSACTIONS tables. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Does Firebird 4 Beta 2 really need libncurses6?
On 29/05/2020 14:33, Alex Peshkoff via Firebird-devel wrote: On 29.05.2020 16:22, Tony Whyman wrote: Just tried to check out Beta 2 on Linux Mint 19.3 (= ubuntu bionic 18.04) and found that the binaries have been compiled to use libncurses6. Ubuntu 18.04 uses libncurses5. You need ubuntu 19.10 if you don't want to go through the bother of backporting libncurses6. By contrast the most recent daily snapshot build has been compiled with libncurses5. Unless there is something really necessary that only libncurses6 provides (and which doesn't affect the snapshot build), this is going to limit the acceptability of Firebird 4. It would be good idea to re-issue the binaries using the same compilation environment as the snapshot builds. Sorry - but sooner of all that means we need to upgrade snapshot build env. We are going to make new version support first of all new distros. Is that really true? If you look at https://packages.ubuntu.com/cgi-bin/search_packages.pl?keywords=libncurses=names=1=breezy=all libncurses5 is supported in the latest Ubuntu releases. I would have thought that while the packages built for a specific OS should use the latest libraries, the general purpose downloads should be conservative in their choice of support packages so that they work on the widest range of systems. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Does Firebird 4 Beta 2 really need libncurses6?
Just tried to check out Beta 2 on Linux Mint 19.3 (= ubuntu bionic 18.04) and found that the binaries have been compiled to use libncurses6. Ubuntu 18.04 uses libncurses5. You need ubuntu 19.10 if you don't want to go through the bother of backporting libncurses6. By contrast the most recent daily snapshot build has been compiled with libncurses5. Unless there is something really necessary that only libncurses6 provides (and which doesn't affect the snapshot build), this is going to limit the acceptability of Firebird 4. It would be good idea to re-issue the binaries using the same compilation environment as the snapshot builds. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Usage of 2020-01-01 as date for some of the time with time zone conversions
I have also been having problems with the use of the TIME WITH TIME ZONE type and have been holding back on posting on the issue until I came up with a suitable proposal. I wrote down my thoughts on the issue a few weeks ago and have appended them to the end of this EMail - at the time of writing the CURRENT_DATE will still used for the conversion - the style used is intended to make it easy to incorporate in the IBX user guide. In the meantime, I have pressed ahead with adding TIME WITH TIME ZONE support to IBX. I have been experimenting with taking the client side conversion to GMT into the IBX code and allowing the user to specify the date at which the conversion to GMT takes place - defaulting to the then FIrebird standard i.e. using the CURRENT DATE. This seems to work well in testing. On 16/05/2020 03:22, Adriano dos Santos Fernandes wrote: On 15/05/2020 12:46, Mark Rotteveel wrote: The decision to use 2020-01-01 as date for some of the time with time zone conversion leads to, in my opinion, confusing behaviour: First, the previous behavior, as you know, is not viable: it broke indexes, foreign keys, unique constraints, stored data. The current behavior is not non-standard, as standard has only offset-based time zones, and with offsets the current behavior is identical to the previous one. Do you have an alternative (that do not broke things), better than use fixed date (as Oracle does with 0001-01-01 and it even more broke behavior) or a recent date? Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Time with Time Zone FB4 introduces the TIME WITH TIME ZONE data type. The TIME data type is a longstanding Firebird data type that records a time in the range 0:00..23:59 and to a precision of 10^-4 seconds. The TIME WITH TIME ZONE data type extends this to include a time zone identifier as part of the value saved in the database. A TIME WITH TIME ZONE data type is always saved in the database as a GMT timestamp plus a time zone id, regardless of whether it was originally input as GMT or as a local time in the context of some time zone. A TIME data type may be used for many purpose: * It can be an elapsed time (< 24 hours) as recorded on a stop watch, or a time code on a video stream. * It can be a time of day with reference to a date stored elsewhere. For example, a master record may include a date and a detailed record may hold a time on that day on which some event occurs alongside information about the event itself. * It can be a scheduled time on some day in the future. This can be any day or a day in a week. For example, shop opening and closing hours may be recorded as a pair of times on a specified day of the week. * It could even be a /sidereal time/. This is defined by wikipedia as a "time scale that is based on Earth's rate of rotation measured relative to the fixed stars". Here a TIME data type could record the sidereal time at which an observation is to be made. Given that a sidereal day is about four minutes shorter than a solar day a separate computation system is needed to convert a scheduled sidereal time to a calendar time. A TIME (without time zone) may be used for any of the above. In the first and last cases, the concept of local time does not apply and hence a TIME WITH TIME ZONE is not of interest. In the second and third cases, prior to FB4, the database designer will have had to have specified whether the time is implicitly recorded in UTC or some local time zone. FB4 allows the time zone to be explicitly added to the value by using a TIME WITH TIME ZONE data type. Inputting a TIME WITH TIME ZONE Data Type The time zone can be given each time a time is entered or set as a session default. Either way, the following applies. Let's say that the time is to be set as 7 am Eastern (New York time). This could be expressed in several ways, each resulting in a different database entry when translated to GMT + time zone id. This is illustrated in the following table. *Original Value * *Time (GMT)* *Time Zone ID* 7:00 EST 12:00 65136 (EST) 7:00 EST5EDT 11:00 or 12:00 65135 (EST5EDT) 7:00 –05:00 12:00 1139 (-05:00) 7:00 America/New York 11:00 or 12:00 America/New_York (65361) The second and fourth cases are somewhat problematic. The time is translated to either 11:00 GMT or 12:00 GMT depending on whether or not daylight savings time applies. For this you either need to assume that the daylight savings time is ignored or apply an assumed date for the translation. Firebird uses the current date when determining whether or not to apply daylight savings time. Hence, 7:00 America/New York will translate to 11:00 GMT during the summer months and to 12:00 GMT during the winter months.
[Firebird-devel] [FB-Tracker] Created: (CORE-6303) Error writing to TIMESTANP/TIME WITH TIME ZONE array
Error writing to TIMESTANP/TIME WITH TIME ZONE array Key: CORE-6303 URL: http://tracker.firebirdsql.org/browse/CORE-6303 Project: Firebird Core Issue Type: Bug Components: Engine Affects Versions: 4.0 Beta 2 Environment: Linux Mint 19.3 amd64 with LI-T4.0.0.1960 Firebird 4.0 Beta 2 Reporter: Tony Whyman I have set up a test of TIME/TIMESTAMP WITH TIME ZONE arrays. The test is failing when putslice is called, with the error message reporting an error writing data to the connection (send_packet/send). The test table is created using: Create Table FB4TestData_ARTZ ( RowID Integer not null, TimeCol TIME WITH TIME ZONE [0:16], TimestampCol TIMESTAMP WITH TIME ZONE [0:16], Primary Key(RowID) ); The test is run by filling each array with 17 values and then writing the array buffer to the server using "putslice". The SDL is identical to a known good SDL block for an array of TIMESTAMP (WITHOUT TIME ZONE) except that the data type is blr_sql_time_tz or blr_timestamp_tz as appropriate. I have seen a connection drop previously when using putslice when the SDL is mis-formatted. However, I can see no problem in this case. The one oddity I can see is that the element size being returned from the metadata is "8" for a TIME WITH TIME ZONE array and "12" for a "TIMESTAMP WITH TIME ZONE" array. Given the data structures, I would have expected "6" and "10". However, I have also checked the metadata for non-array TIME WITH TIME ZONE and TIMESTAMP WITH TIME ZONE columns and these also return "8" and "12" respectively. If I execute a 'SET BIND OF TIME ZONE TO EXTENDED' statement, then the non-array column sizes remain as "8" and "12" even though the data types have changed to SQL_TIME_TZ_EX and SQL_TIMESTAMP_TZ_EX, respectively. I presume that field alignment is being forced to a four byte boundary. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6302) Error writing an array of NUMERIC(24, 6) to the database when using FB4 Development Snapshot
Error writing an array of NUMERIC(24,6) to the database when using FB4 Development Snapshot --- Key: CORE-6302 URL: http://tracker.firebirdsql.org/browse/CORE-6302 Project: Firebird Core Issue Type: Bug Components: Engine Affects Versions: 4.0 Beta 1 Environment: Using the FB4 Development Snapshot under Linux Mint 19.3 AMD64 Reporter: Tony Whyman I have created a table as follows in order to test out FB4 array handling with the new datatypes. Create Table FB4TestData_DECFloat_AR RowID Integer not null, Float16 DecFloat(16) [0:16], Float34 DecFloat(34) [0:16], BigNumber NUMERIC(24,6) [0:16], Primary Key(RowID) ); The metadata indicates that the first two arrays are arrays of DecFloat(16) and DecFloat(34) respectively, while the latter is an INT128 array with a scale factor of -6. Running separate tests on each array column: the first two perform as expected with both read and write operations successful and the read results corresponding to the write. However, the "putslice" API method fails when writing the NUMERIC(24,6) array type with the error message: column not array or invalid dimensions (expected 0, encountered 1). The SDL for the putslice is the same as for the Float34 array except for the data type and scale factor. The SDL is generated following src/yvalve/array.epp -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Investigating Array types and Firebird 4
I have been investigating how the current development snapshot works with arrays and the new Firebird 4 data types. So far, the I have the following results: 1. DECFLOAT(16): works fine. Able to declare an array (e.g. Float16 DecFloat(16) [0:16]), write to array elements, save the row and read back the results. 2. DECFLOAT(32): works fine. Able to declare an array (e.g. Float34 DecFloat(34) [0:16]), write to array elements, save the row and read back the results. 3. NUMERIC(24,6) [0:16] somewhat surprisingly results in an array of INT128 elements. When attempting to write the array to the database, I get the error message: "column not array or invalid dimensions (expected 0, encountered 1)". Some uncertainty over where the SDL block should include a scale, but its absence or presence does not seem to make a difference. 4. TIME WITH TIME ZONE [0:16]: Able to declare an array (e.g. TimeCol TIME WITH TIME ZONE [0:16]). However, when saving to the database, the connection drops with a send packet/send error. 5. TIMESTAMP WITH TIME ZONE [0:16]: as above. The results are thus patchy. The lack of SDL documentation makes it difficult to know whether the above issues are due to SDL errors on my side or bugs in the server. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB4 linux snapshot build appears to be broken
Panic over. I had libtomcrypt0 installed and masquerading as libtomcrypt1. Correct version now installed and FB4 snapshot working. On 08/05/2020 14:16, Tony Whyman wrote: On 08/05/2020 14:07, Alex Peshkoff via Firebird-devel wrote: On 2020-05-08 16:00, Tony Whyman wrote: /opt/firebird4b2/plugins/libChaCha.so: undefined symbol: chacha_setup On 08/05/2020 13:54, Alex Peshkoff via Firebird-devel wrote: On 2020-05-08 15:43, Tony Whyman wrote: Finally got round to working out why there was a problem. If I change the Wirecrypt setting in firebird.conf to #WireCryptPlugin = ChaCha, Arc4 WireCryptPlugin = Arc4 then everything works fine. I have downloaded today's snapshot and the same problem is present i.e. the wirecrypt plugin raises an exception unless the above change is made. What exception? Is there a missing dependency for the ChaCha plugin? If I run ldd on libChaCha.so I get ldd libChaCha.so linux-vdso.so.1 (0x7ffcd5bb6000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f5d75f99000) libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x7f5d75d76000) libfbclient.so.2 => /opt/firebird4b2/plugins/./../lib/libfbclient.so.2 (0x7f5d75859000) libtommath.so.0 => /usr/lib/libtommath.so.0 (0x7f5d75642000) libtomcrypt.so.1 => /usr/lib/x86_64-linux-gnu/libtomcrypt.so.1 (0x7f5d7538c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f5d74fee000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f5d74dd6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f5d74bb7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f5d747c6000) /lib64/ld-linux-x86-64.so.2 (0x7f5d764f2000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x7f5d7459c000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x7f5d7431b000) You see yourself - no missing dependencies. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Your /usr/lib/x86_64-linux-gnu/libtomcrypt.so.1 appears to be built w/o chacha support. Try to uninstall tomcrypt package from your OS (btw, what kind of linux is it) and reinstall firebird after it. It's an up-to-date Linux Mint 19.3 (aka ubuntu (bionic) 18.04). I would not have thought it was much different in respect of libtomcrypt from any other Debian derivative. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB4 linux snapshot build appears to be broken
On 08/05/2020 14:07, Alex Peshkoff via Firebird-devel wrote: On 2020-05-08 16:00, Tony Whyman wrote: /opt/firebird4b2/plugins/libChaCha.so: undefined symbol: chacha_setup On 08/05/2020 13:54, Alex Peshkoff via Firebird-devel wrote: On 2020-05-08 15:43, Tony Whyman wrote: Finally got round to working out why there was a problem. If I change the Wirecrypt setting in firebird.conf to #WireCryptPlugin = ChaCha, Arc4 WireCryptPlugin = Arc4 then everything works fine. I have downloaded today's snapshot and the same problem is present i.e. the wirecrypt plugin raises an exception unless the above change is made. What exception? Is there a missing dependency for the ChaCha plugin? If I run ldd on libChaCha.so I get ldd libChaCha.so linux-vdso.so.1 (0x7ffcd5bb6000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f5d75f99000) libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x7f5d75d76000) libfbclient.so.2 => /opt/firebird4b2/plugins/./../lib/libfbclient.so.2 (0x7f5d75859000) libtommath.so.0 => /usr/lib/libtommath.so.0 (0x7f5d75642000) libtomcrypt.so.1 => /usr/lib/x86_64-linux-gnu/libtomcrypt.so.1 (0x7f5d7538c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f5d74fee000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f5d74dd6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f5d74bb7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f5d747c6000) /lib64/ld-linux-x86-64.so.2 (0x7f5d764f2000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x7f5d7459c000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x7f5d7431b000) You see yourself - no missing dependencies. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Your /usr/lib/x86_64-linux-gnu/libtomcrypt.so.1 appears to be built w/o chacha support. Try to uninstall tomcrypt package from your OS (btw, what kind of linux is it) and reinstall firebird after it. It's an up-to-date Linux Mint 19.3 (aka ubuntu (bionic) 18.04). I would not have thought it was much different in respect of libtomcrypt from any other Debian derivative. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB4 linux snapshot build appears to be broken
/opt/firebird4b2/plugins/libChaCha.so: undefined symbol: chacha_setup On 08/05/2020 13:54, Alex Peshkoff via Firebird-devel wrote: On 2020-05-08 15:43, Tony Whyman wrote: Finally got round to working out why there was a problem. If I change the Wirecrypt setting in firebird.conf to #WireCryptPlugin = ChaCha, Arc4 WireCryptPlugin = Arc4 then everything works fine. I have downloaded today's snapshot and the same problem is present i.e. the wirecrypt plugin raises an exception unless the above change is made. What exception? Is there a missing dependency for the ChaCha plugin? If I run ldd on libChaCha.so I get ldd libChaCha.so linux-vdso.so.1 (0x7ffcd5bb6000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f5d75f99000) libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x7f5d75d76000) libfbclient.so.2 => /opt/firebird4b2/plugins/./../lib/libfbclient.so.2 (0x7f5d75859000) libtommath.so.0 => /usr/lib/libtommath.so.0 (0x7f5d75642000) libtomcrypt.so.1 => /usr/lib/x86_64-linux-gnu/libtomcrypt.so.1 (0x7f5d7538c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f5d74fee000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f5d74dd6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f5d74bb7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f5d747c6000) /lib64/ld-linux-x86-64.so.2 (0x7f5d764f2000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x7f5d7459c000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x7f5d7431b000) You see yourself - no missing dependencies. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB4 linux snapshot build appears to be broken
Finally got round to working out why there was a problem. If I change the Wirecrypt setting in firebird.conf to #WireCryptPlugin = ChaCha, Arc4 WireCryptPlugin = Arc4 then everything works fine. I have downloaded today's snapshot and the same problem is present i.e. the wirecrypt plugin raises an exception unless the above change is made. Is there a missing dependency for the ChaCha plugin? If I run ldd on libChaCha.so I get ldd libChaCha.so linux-vdso.so.1 (0x7ffcd5bb6000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f5d75f99000) libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x7f5d75d76000) libfbclient.so.2 => /opt/firebird4b2/plugins/./../lib/libfbclient.so.2 (0x7f5d75859000) libtommath.so.0 => /usr/lib/libtommath.so.0 (0x7f5d75642000) libtomcrypt.so.1 => /usr/lib/x86_64-linux-gnu/libtomcrypt.so.1 (0x7f5d7538c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f5d74fee000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f5d74dd6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f5d74bb7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f5d747c6000) /lib64/ld-linux-x86-64.so.2 (0x7f5d764f2000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x7f5d7459c000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x7f5d7431b000) On 30/04/2020 17:09, Alex Peshkoff via Firebird-devel wrote: On 2020-04-30 12:10, Tony Whyman wrote: All I get when trying to use the current snapshot and opening a database is the error message "Invalid Clumplet buffer structure: path length doesn't match with clumplet". I am using a test program that works with all previous versions of Firebird. If I try to use ISQL with the example employee database, I get bin/isql: symbol lookup error: /opt/firebird4b2/plugins/libChaCha.so: undefined symbol: chacha_setup when I try to open the database using isql -user SYSDBA -pass masterkey localhost:employee Today snapshot works for me. # ./isql localhost:employee -user sysdba -password 'masterkey' -z ISQL Version: LI-T4.0.0.1948 Firebird 4.0 Beta 2 Server version: LI-T4.0.0.1948 Firebird 4.0 Beta 2 LI-T4.0.0.1948 Firebird 4.0 Beta 2/tcp (fbs3)/P16:C LI-T4.0.0.1948 Firebird 4.0 Beta 2/tcp (fbs3)/P16:C Database: localhost:employee, User: SYSDBA SQL> show db; Database: localhost:employee Owner: SYSDBA PAGE_SIZE 8192 # Wire crypt plugin: ChaCha Default Character set: NONE SQL> Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] SET BIND OF TIME ZONE TO EXTENDED and Data Input
In the README.time_zone, the SET BIND OF TIME ZONE TO EXTENDED is described as being intended to "solve a problem of representing correct time on clients missing ICU library". All the text I can find, discusses how this is used when reading a TIMESTAMP WITH TIME ZONE data type from a database, but how does this apply to data input? My understanding is that when a client local ICU is present, the encodeTimeStampTz call uses the client local ICU to translate the timestamp to GMT before passing it to the database. If no client local ICU is present and the SET BIND has been executed then what happens? Is the timestamp passed to the server where it is translated to GMT or is an error raised? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] FB4 linux snapshot build appears to be broken
All I get when trying to use the current snapshot and opening a database is the error message "Invalid Clumplet buffer structure: path length doesn't match with clumplet". I am using a test program that works with all previous versions of Firebird. If I try to use ISQL with the example employee database, I get bin/isql: symbol lookup error: /opt/firebird4b2/plugins/libChaCha.so: undefined symbol: chacha_setup when I try to open the database using isql -user SYSDBA -pass masterkey localhost:employee Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU File Deployment Guidance
On 29/04/2020 13:59, Adriano dos Santos Fernandes wrote: There probably should thus be a firebird-tzdata package as part of the set of (deb/rpm) Firebird packages containing the latest tzdata files. It should then be easy enough for the package maintainer to roll out updates to the time zone database through the usual channels. That would be good. Some one also needs to fix the daily snapshot Linux build for FB4 to include the tzdata directory. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU File Deployment Guidance
On 29/04/2020 13:27, Adriano dos Santos Fernandes wrote: On 29/04/2020 08:37, Paul Reeves wrote: On Wed, 29 Apr 2020 11:59:43 +0100 Tony Whyman wrote: Hopefully, an installer package will eventually be made available to automate the process. However, the above manual procedure is all that is currently available. I'm wondering how people think we could automate this process. Thoughts that immediately come to mind are: - We have no control over the release cycle of tzdata. The update to https://github.com/FirebirdSQL/firebird/tree/master/extern/icu/tzdata seems to be manual at the moment. Looks like you are not aware of things, where automated daily github action peeks changes in https://github.com/unicode-org/icu-data/tree/master/tzdata/icunew and creates a pull request for us. It has just happened this week. Adriano My own view is that a Windows Installer Package for tzdata is essential if the time zone database is to be kept up-to-date. Recall that most operational deployments are highly controlled production environments where the System Administrator will demand that updates are strictly controlled and subject to source verification. It also should be possible to quickly roll back any updates if a problem is found. If the pull request is automated then it should not be difficult to rebuild the tzdata installation package automatically and, once it has been tested, it can be formally released with an EMail announcement sent out to alert interested parties. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU File Deployment Guidance
On 29/04/2020 13:22, Adriano dos Santos Fernandes wrote: This has nothing to do with ICU. There is no ICU tz data package in Ubuntu. The data is with the libicu package inside the libraries. OK. That seems to be supported by the ICU documentation. There probably should thus be a firebird-tzdata package as part of the set of (deb/rpm) Firebird packages containing the latest tzdata files. It should then be easy enough for the package maintainer to roll out updates to the time zone database through the usual channels. However, Windows is still a problem. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU File Deployment Guidance
On 29/04/2020 12:12, Adriano dos Santos Fernandes wrote: In Linux (Ubuntu as least) and probably MacOS, the ICU time zone database is not maintained updated in an Ubuntu release. I guess even if they rebuild and update the library, it still comes with the original tz database. So the process of update tz data with *.res files should be for all platforms. I don't claim any expertise on linux time zone files. However, there is a ubuntu package "tzdata" which claims to include all the timezone files. I am currently using Linux Mint 19.3 (ubuntu 18.04) and the most recent changelog entry for my version of the tzdata package is: tzdata (2019c-0ubuntu0.18.04) bionic; urgency=medium * New upstream version, affecting the following future timestamps: - Fiji's next DST transitions will be 2019-11-10 and 2020-01-12 instead of 2019-11-03 and 2020-01-19. - Norfolk Island will observe Australian-style DST starting in spring 2019. The first transition is on 2019-10-06. -- Adam Conrad Fri, 20 Sep 2019 03:11:15 -0600 which seems pretty recent to me. This is also the current version for 16.04 and 19.04. Checking the ubuntu package webpage, it looks like 2019c-3 has been rolled out to 19.10. Although the changelog suggest nothing important in the minor patch. The most recent Debian version is 2020a-1 (released 4 days ago). so I'll be interested to see how quickly that feeds down to ubuntu 18.04 - if ever. Its changelog is tzdata (2020a-1) unstable; urgency=medium * New upstream version, affecting the following future timestamps: - Morocco springs forward on 2020-05-31, not 2020-05-24. - Canada's Yukon advanced to -07 year-round on 2020-03-08. * Bump Standards-Version to 4.5.0 (no changes). -- Aurelien Jarno Fri, 24 Apr 2020 21:32:19 +0200 I would certainly prefer to rely on the Ubuntu update cycle for the tzdata rather than a manual update and the above suggests that this should be possible. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] ICU File Deployment Guidance
Many thanks to those who responded yesterday to my question on ICU library file installation under Windows. I am in the process of updating the IBX User Manual to include the Firebird 4 extensions and this question arose when updating the deployment guidelines. Specifically, giving IBX users guidance on when to include and provide an update mechanism for the time zone database. it is important to get this guidance right as the target reader is a developer using IBX and Firebird but who does not intend to become an expert in Firebird installation. Standalone clients and embedded server installations are the main focus. The draft text for the IBX deployment guidelines for the ICU files follows. I would be grateful for any review comments that can point to any remaining mis-understandings I may have about how the time zone database is used and deployed in Firebird 4. == The ICU Files For character set management including collation sequences, Firebird uses an external code library - International Components for Unicode” (ICU). Firebird 4 and later also uses this code library as the source of its Time Zone Database. This is used in support of TIME/TIMESTAMP WITH TIME ZONE data types and to convert local times to and from GMT taking into account the time zone and any daylight savings time offsets that need to be applied. The ICU library is deployed as a DLL or Shared Object (.so) and may be provided as part of the Operating System. * For Linux distros, the ICU shared objects are always deployed by the distro and are kept up-to-date as part of the normal OS update cycle. * For Microsoft Windows, the ICU DLLs have been included in the Windows OS from Windows 10 Version 1703 (Creators Update) onwards. They are not present in earlier versions of Windows. However, Firebird will always ignore the Windows ICU files and will instead use the ICU files installed with the database and located in the Firebird installation folder. * For macOS, the ICU shared objects are provided as part of macOS. The ICU library needs to be up-to-date in order to correctly translate local times to and from GMT. This is because, from time to time, there are legislative changes to time zones and daylight savings times and these need to recorded in the ICU library. /Note that as the ICU libraries are also used for character set collation sequences, an updated ICU library can also include a change to character set collations and thus may require that any indexes that depend upon an updated collation sequence have to be rebuilt – or a gbak backup/restore cycle is used to rebuild the indexes./ Updating the Time Zone Database under Linux or macOS This is performed automatically when an OS update is performed and the update contains an updated ICU. Most Linux distros can usually be relied upon to role out updates to time zone databases in a timely manner. Updating the Time Zone Database under Windows Each Firebird incremental release includes the most up-to-date version of the time zone database when it is released. However, the time zone database may also need to be updated between Firebird releases. This is not performed automatically under Windows and has to be manually initiated. Under Windows and in order to avoid having to update the time zone database without also needing a full ICU library update, a copy of the time zone database files are held in the \tzdata folder, as a set of '*.res' files. These are used in preference to the time zone database in the ICU DLLs. The time zone database can be updated by simply replacing these files. When new versions of the time zone database files are released, they are made available at: https://github.com/FirebirdSQL/firebird/tree/master/extern/icu/tzdata The file “le.zip” can be downloaded from this page and contains the replacement '*.res' files for use on little endian architectures (e.g. Intel and AMD64 architectures). These have to extracted and the current versions in the \tzdata folder replaced with the updated versions from the zip. Hopefully, an installer package will eventually be made available to automate the process. However, the above manual procedure is all that is currently available. The time zone database for both Firebird Server and embedded server installations should be kept up-to-date and the source of the time zone database files regularly checked for updates. Client Side Considerations For character set collations, only the Firebird server/embedded server needs access to the ICU library. However, by default, Firebird clients also require access to the ICU library (or tzdata files) for time zone information. In Firebird 4 Beta 1 client local access to the ICU library was mandatory. However, in later versions, this is optional and it is possible for a Firebird client to rely on the server for all time zone computations. This mode
Re: [Firebird-devel] Installing the ICU Files under Windows
So, I guess this has been added since beta 1. I can see it in the latest Windows snapshot, but not in the latest Linux snapshot. So the README should make clear that the tzdata directory only applies to the Windows builds? Back to my original concern. While the README file gives an outline of what is necessary, this is not for the end user. Someone has got to produce an installer package to update the tzdata directory every time the time zone database is updated. Correct? Note that you cannot ask every end user to update Firebird client installation by copying a file into a potentially protected folder on a locked down laptop. There has to be some sort of official installer. You are getting a too old beta/snapshot. See the most recent snapshot. tzdata directory should be in the binary kits. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Installing the ICU Files under Windows
On 28/04/2020 15:11, Adriano dos Santos Fernandes wrote: On 28/04/2020 11:05, Tony Whyman wrote: On 28/04/2020 14:59, Adriano dos Santos Fernandes wrote: https://github.com/FirebirdSQL/firebird/blob/master/doc/sql.extensions/README.time_zone.md#updating-the-time-zone-database This text applies to building firebird. You are wrong, maybe you should read again. Adriano If I am wrong, it may be because the instructions are not clear. The README file says: "The content of the zip file must be extracted in the |tzdata| subdirectory of Firebird's root, overwriting the others |*.res| files of the old database. Note: |/tzdata| is the default directory where Firebird looks for the database. It could be overriden with the |ICU_TIMEZONE_FILES_DIR| environment variable." I see an icu/tzdata directory in the source tree, but I don't see one in either the Windows zip or the Linux archive. 1. Does this directory only exist when created by the end user when a time zone database update is installed or should it be created when Firebird is installed? 2. Is this a procedure that the local database adminstrator has to carry out as part of Firebird maintenance? 3. Does this apply to Linux distributions as well as to Windows distributions? 4. Do I have to perform the same procedure on every database client so that their local ICU copies are also up-to-date? This all goes back to my original worry: what do I tell the end user when it comes to maintaining the Firebird server's time zone database? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Installing the ICU Files under Windows
On 28/04/2020 14:59, Adriano dos Santos Fernandes wrote: https://github.com/FirebirdSQL/firebird/blob/master/doc/sql.extensions/README.time_zone.md#updating-the-time-zone-database This text applies to building firebird. I am worrying about the end user's strategy. My point is that there will need to be incremental releases of Firebird for each ICU update needed to reflect time zone changes. That is unless the ICU is separately packaged. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Installing the ICU Files under Windows
On 28/04/2020 14:47, Mark Rotteveel wrote: On 28-04-2020 15:43, Tony Whyman wrote: Apologies. I forgot to mention that I was worrying about using the ICU for the time zone database rather than character sets. Why does that worry you? Mark Because my understanding is that the ICU is the source of the time zone database and will have to be updated every time a time zone changes or there are changes to the dates for daylight savings time coming into effect. Here's one of many examples: https://www.timeanddate.com/news/time/eu-scraps-dst.html I was worrying about recommending a strategy for updating the ICU files under Windows. So far I also seem to have learnt: 1. There is no intent to use the Windows 10 ICU library in Firebird. 2. There is a risk of a stealth (ICU) upgrade under Linux if the upgrade affects indexes that depend upon character set collation sequences. Neither of which I expected. For Windows, there may need to Firebird incremental updates to catch up with time zone changes. For Linux, this is automatic - but beware of index problems... Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Installing the ICU Files under Windows
Apologies. I forgot to mention that I was worrying about using the ICU for the time zone database rather than character sets. On 28/04/2020 14:39, Mark Rotteveel wrote: On 28-04-2020 15:28, Tony Whyman wrote: On 28/04/2020 14:22, Adriano dos Santos Fernandes wrote: Databases (indexes) depends on ICU being the same (same sort key version) or they need to be backed-up and restored (with gbak) to be fully functional. That surprises me, as I thought that a TIMESTAMP WITH TIME ZONE was stored as a GMT time plus a time zone id. Why should this be ICU dependent - or does the time zone id change between ICU versions? This isn't about time zones, but about ICU also being used for other things (character set collations), where updates of ICU might require an index rebuild for it to work. Mark Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Installing the ICU Files under Windows
On 28/04/2020 14:22, Adriano dos Santos Fernandes wrote: I advise not to do that same thing in Windows as done in Linux, using the OS ICU. In Linux situation is less complex, it's a server OS. I also think it was an incorrect thing started without notice by the distro people. So is the ICU used differently in Linux and Windows? Could a problem arise if I update my Linux distro and the ICU gets updated? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Installing the ICU Files under Windows
On 28/04/2020 14:22, Adriano dos Santos Fernandes wrote: Databases (indexes) depends on ICU being the same (same sort key version) or they need to be backed-up and restored (with gbak) to be fully functional. That surprises me, as I thought that a TIMESTAMP WITH TIME ZONE was stored as a GMT time plus a time zone id. Why should this be ICU dependent - or does the time zone id change between ICU versions? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Installing the ICU Files under Windows
I have noticed that the FB4 snapshot builds for Windows include the icu files icudt52.dll, etc. They probably should not be installed for Windows 10 Version 1703 (Creators Update) onwards. Does the Windows installer make such a check and only install the icu files for older versions of Windows? Alternatively, does the Firebird code that loads the dll prefer the Windows distribution (i.e. icuuc.dll and icuin.dll) over any ICU files in its local folder? One reason that I could not check this is that I couldn't find any Windows Installer source files in the source code distribution. Does Firebird use Open Source WIX XML files for the installer or some proprietary tool to create the .msi file? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Firebird API Incompleteness
While on the subject of incompleteness in the Firebird 3 API, I never found any alternative to the following utility functions from the legacy API. isc_sqlcode isc_sql_interprete isc_interprete isc_event_counts isc_event_block isc_free This is not a big deal as these are all local functions with no database context. The first 3 are needed for decoding the status vector and could be included in IStatus, while the latter 3 are needed to manage the events buffer and hence could be part of IEvents. Was there any reason for leaving these out of the new API? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB 3 & 4, new API and the ARRAY type
Well, it comes as news to me that the array type is unsupported in the new API. IBX supports the array type with the new FB3 api and it all seems to work fine. The example demos work with both Firebird 2.5 and Firebird 3.0.To do this, I had to reverse engineer the FB 2.5 code in order to understand the param block structure, but once I had done that, it all worked! You can check out he IB code to see how it's done. It uses getslice and putslice and manages its own descriptors.I see no reason to deprecate the array API unless it breaks something else. Original message From: Pavel Cisar Date: 22/04/2020 12:49 (GMT+00:00) To: firebird-devel@lists.sourceforge.net Subject: [Firebird-devel] FB 3 & 4, new API and the ARRAY type Hi all,I'm writing new Python driver built using new FB3 API (mainly to get support for new features not available through old API). I was surprised that ARRAY type support in new API is incomplete and thus it's not possible to handle this data type in drivers/applications. Specifically, the API provides getSlice & putSlice methods on IAttachment, but there isn't any equivalent for very important isc_array_lookup_bounds function (isc_array_lookup_desc and isc_array_set_desc are also missing, but they are not important for driver developers).Alex explained to me, that array support is not functional in new API, because core developers are considering to deprecate & remove ARRAY type support from future Firebird versions. And because the final decision was not made yet, the arrays support in new API get stuck in incomplete state.Personally, I have no problem with deprecation of the ARRAY type, as this type is mostly not used (if at all) by Firebird users. Also, I understand the reasoning for missing methods that would pollute the interfaces with methods that would become eventually obsolete. However, I'm really stunned how this issue was handled.While it's ok to provide only new API for new features (like scrollable cursors), the NEW API SHOULD support all old Firebird features present in given version and available through old API. And ARRAY type is such a feature. Although it's probably not used much by Firebird users, it's still used in example EMPLOYEE database that's used a lot in books, articles etc., and support for it is available to end users via some drivers (like FDB Python driver) for many years. So it's definitely POSSIBLE, that users will run into situations when lack of ARRAY support will be at least awkward (missing output), if not straight fatal.Argument that one could always use old API to access ARRAYs will not stand, because some new features are not available through it, and they could not be used together. Users should NOT be forced to choose between two API's with distinct features, at least one should be the superset (preferably the new one).Also, if ARRAY type should be ever removed from Firebird, it should be done via proper deprecation process (that also includes creation of new example database without it). Hence I consider the lack of (at least) isc_array_lookup_bounds equivalent in new API as a serious BUG, that should be fixed in Firebird 3 & 4. As driver developer I would really appreciate if it would be fixed for 3.0.6 due in June.While ARRAY support is the pressing matter here, there are other old API functions that does not have new API replacements (for example isc_blob_lookup_desc, and many others). I think that it's the good time to compare old and new API's, and create a document that will contain table listing the Old API functions with their New API counterparts, or explanation why it was decided to not provide such equivalent in new API. Such document should be first discussed here (so we could decide whether to fix more new API bugs asap), and then become a part of Firebird documentation set (as driver developer, I was really surprised that such document does not exists yet, as it's invaluable for porting from old to new API).best regardsPavel CisarFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-develFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] IUtile API Version numbering
Yes, the "missing types" problem was fixed earlier today. ISC_TIMESTAMP_TZ_EX ISC_SHORT ISC_TIME_TZ_EX were all missing from Firebird.pas. It should now compile correctly. On 19/03/2020 15:38, Norbert Saint Georges wrote: Tony Whyman a écrit : See http://tracker.firebirdsql.org/browse/CORE-6207 this is the difference between the compilation of the day before yesterday and that of the git less than an hour ago Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] IUtile API Version numbering
See http://tracker.firebirdsql.org/browse/CORE-6207
On 19/03/2020 15:27, Norbert Saint Georges wrote:
for information, the differential from yesterday's "firebird.pas"
139c139,140
< ISC_USHORT = word; { 16 bit unsigned }
---
ISC_USHORT = word; { 16 bit unsigned }
ISC_SHORT = smallint; { 16 bit signed }
145a147,152
ISC_TIME_TZ_EX = record
utc_time: ISC_TIME;
time_zone: ISC_USHORT;
ext_offset: ISC_SHORT;
end;
153a161,166
end;
ISC_TIMESTAMP_TZ_EX = record
utc_timestamp: ISC_TIMESTAMP;
time_zone: ISC_USHORT;
ext_offset: ISC_SHORT;
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] IUtile API Version numbering
I have been paying attention recently to the VERSION number for the IUtil API, and I can't see the logic behind the version numbering scheme. I am looking at the Firebird.pas file. In Firebird 3, I see "const VERSION = 13;" In Firebird 4 Beta 1, I see "const VERSION = 21;" In the most recent development source (downloaded using git clone https://github.com/FirebirdSQL/firebird.git), and after building Firebird, I see "const VERSION = 4;" The first two do look like some kind of progression. However, the current build VERSION seems to have no logic to it. Can someone enlighten me as to the logic behind the version numbers. This is an issue with IUtil as additional functions have been added to the interface and it is useful to be able to have a simple means of determining the available function set. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Introduction of EXTENDED TIME/TIMESTAMP WITH TIME ZONE?
On 10/03/2020 10:34, Dimitry Sibiryakov wrote: 10.03.2020 09:48, Alex Peshkoff via Firebird-devel wrote: May be for old windows versions - yes, some problems. All the others upgrade ICU as a part of regular OS upgrade. I.e. if one already has local ICU extended format is definitely useless overhead. ICU appeared only in recent builds of Windows 10. Windows 7-8 and servers don't have it. Some Linux distros package ICU in a very strange way with names that Firebird don't know how to use. The definitive document from Microsoft is here: https://docs.microsoft.com/en-us/windows/win32/intl/international-components-for-unicode--icu- It looks like the icu dlls are only available from Version 1709 onwards. Interestingly, they are called: icuuc.dll, and icuin.dll in true MS fashion, the filenames do not include a version number. Looking at https://github.com/fyatao/firebird/blob/master/src/common/unicode_util.cpp it looks like Firebird will try and load icu files with the version number in them i.e. using the template #if defined(WIN_NT) const char* const inTemplate = "icuin%s.dll"; const char* const ucTemplate = "icuuc%s.dll"; I may have missed it, but I can't see the code that also tries a missing version number for the dll file name. i.e. it won't try to load the MS supplied versions. Firebird 4 Beta 1 was distributed with icuin63.dll and icuuc63.dll and I would expect the server to always load this version regardless of the ICU version provided with Windows 10. On Windows, Firebird should really try and load the MS supplied ICU dlls first and then only try and load the Firebird distributed ones afterwards. If my reading of the code is correct then, Firebird will only ever load the Firebird distributed versions - and this looks like a bug. I apologise in advance if I mis-read the code. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Introduction of EXTENDED TIME/TIMESTAMP WITH TIME ZONE?
On 04/03/2020 17:16, Alex Peshkoff via Firebird-devel wrote: On 2020-03-04 20:01, Mark Rotteveel wrote: What is the purpose of introducing yet another datatype called EXTENDED TIME/TIMESTAMP WITH TIME ZONE? Help users missing ICU on the client work with time zones. I am still trying to get my head around this new feature. As I understand it, it exists for support of broken or legacy systems without an ICU shared library. That is, following a "SET BIND..." the server's ICU is used instead of the client's. My reading is that when the server's ICU is used, TIMESTAMP/TIME with TIME ZONE values are returned as UTC with both the time zone id and the time zone offset as computed by the server's ICU taking into account any daylight savings time variations. I presume that in extended mode, when a TIMESTAMP/TIME with TIME ZONE is used as a statement parameter, the value is passed to the server as a local time plus the time zone id and the server's ICU is used to convert this to UTC before evaluating the query. From the user's point of view, the result is the same. There's just a small extra overhead with the wire protocol in terms of packet size, but the number of packets is the same. The question then arises as to why this is not the normal way of working? Using the client's local ICU introduces a maintenance headache. If it is out-of-step with the server (or other clients) then inconsistent results may occur when computing daylight savings time offsets. So why shouldn't I just always call "SET BIND..." as soon as a database connection is opened, support only the EXTENDED TIME/TIMESTAMP WITH TIME ZONE, and avoid the risk of ICU's getting out of step? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Introduction of EXTENDED TIME/TIMESTAMP WITH TIME ZONE?
This is all post Beta 1 stuff. When will there be a Beta 2 release incorporating EXTENDED TIME(STAMP) WITH TIME ZONE? On 06/03/2020 09:48, Alex Peshkoff via Firebird-devel wrote: On 2020-03-06 12:43, Mark Rotteveel wrote: I'll grudgingly implement support in Jaybird by handling it identical to a normal TIMESTAMP WITH TIME ZONE, just in case someone configures a bind to EXTENDED TIME(STAMP) WITH TIME ZONE. Absolutely right solution. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird UDF compiled with Lazarus 2.0.0 and FPC 3.0.4
My guess is that you are using the wrong calling convention for Linux.
Note: IBX uses "stdcall" when accessing the Firebird API under Windows,
but changes to using "cdecl" under Linux.
On 26/02/2020 13:30, Massimo Fazzolari wrote:
Hi,
I compiled a Firebird UDF library written in object pascal. It works
perfectly well on Windows.
On Linux I get this error:
SQL> select SIMPLE('test') from client;
Statement failed, SQLSTATE = 08006
Error reading data from the connection.
Firebird Version: Classic 2.5.9.27139-0
Lazarus Version: 2.0.0
Free Pascal Compiler: 3.0.4
These are the compiler options used:
/usr/bin/fpc
-Tlinux
-Px86_64
-MObjFPC
-Scghi
-Cg
-O1
-Xs
-l
-vewnhibq
-Filib/x86_64-linux
-Fl/opt/firebird/lib
-Fuextra
-Fuextra/units
-Fu/usr/share/lazarus/2.0.0/lcl/units/x86_64-linux
-Fu/usr/share/lazarus/2.0.0/components/lazutils/lib/x86_64-linux
-Fupackages/HashLib/src/Packages/FPC/lib/HashLib4Pascal/x86_64-linux
-Fu/usr/share/lazarus/2.0.0/packager/units/x86_64-linux
-Fu.
-FUlib/x86_64-linux
-FE.
-olibsitaudflib.so
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6243) Firebird 4 Beta 1 rejects SQL 2003 compliant CREATE TRIGGER syntax
Firebird 4 Beta 1 rejects SQL 2003 compliant CREATE TRIGGER syntax -- Key: CORE-6243 URL: http://tracker.firebirdsql.org/browse/CORE-6243 Project: Firebird Core Issue Type: Bug Components: Engine Affects Versions: 4.0 Beta 1 Environment: Linux Mint 19.3 64-bit Firebird binary installed from firebirdsql.org Reporter: Tony Whyman In a simple test of the Firebird 4 engine, I dumped the example employee database (from a Firebird 3 source) and edited the first CREATE TRIGGER to: CREATE TRIGGER SET_CUST_NO ACTIVE BEFORE INSERT POSITION 0 ON CUSTOMER AS BEGIN if (new.cust_no is null) then new.cust_no = gen_id(cust_no_gen, 1); END ^ so that it was SQL 2003 compliant. When feeding the SQL back in to create a new database using the Firebird 4 isql, this caused the following error message: Statement failed, SQLSTATE = 42000 Dynamic SQL Error -SQL error code = -104 -Token unknown - line 2, column 22 -POSITION I also fed exactly the same script back to a Firebird 3 Server/isql and the script completed with no errors. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] PGP keys, software security, and much more threatened by new SHA1 exploit
Hence why I reported http://tracker.firebirdsql.org/browse/CORE-5788 and why SRP256 has been available from Firebird 3.0.4 onwards. On 09/01/2020 17:25, marius adrian popa wrote: https://arstechnica.com/information-technology/2020/01/pgp-keys-software-security-and-much-more-threatened-by-new-sha1-exploit/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6207) Modifications needed to complile Firebird.pas under FPC
Modifications needed to complile Firebird.pas under FPC
---
Key: CORE-6207
URL: http://tracker.firebirdsql.org/browse/CORE-6207
Project: Firebird Core
Issue Type: Bug
Components: API / Client Library
Affects Versions: 4.0 Beta 1
Environment: Linux Mint 18.3, FPC 3.0.4
Reporter: Tony Whyman
Priority: Blocker
This bug report is intended to record the changes to Firebird.pas needed for it
to successfully compile under FPC 3.0.4. This is the result of preparation work
for upgrading IBX for Lazarus to use new Firebird 4 features.
1. The reserved word "record" is used as a parameter name and needs to be
escaped i.e. to "".
2. Unknown Types:
Replace:
isc_tr_handle = ^integer32;
isc_stmt_handle = ^integer32;
with
isc_tr_handle = ^FixedInt;
isc_stmt_handle = ^FixedInt;
3. Missing Types:
ISC_USHORT = word; { 16 bit unsigned }
ISC_TIME_TZ = record
utc_time: ISC_TIME;
time_zone: ISC_USHORT;
end;
ISC_TIMESTAMP = record
timestamp_date: ISC_DATE;
timestamp_time: ISC_TIME;
end;
ISC_TIMESTAMP_TZ = record
utc_timestamp: ISC_TIMESTAMP;
time_zone: ISC_USHORT;
end;
4. Remove or comment out:
function fb_get_master_interface : IMaster; cdecl; external 'fbclient';
This is not an appropriate external function declaration for a dynamic link
library and only applies to a static library.
When testing with FPC, if this line was not commented out, a linker error
occurs when trying to link with the debug heap
manager (needed for testing for memory leaks).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://tracker.firebirdsql.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] IResultSet and IStatement releasing
IBX's Firebird Pascal API uses the IProvider interface to get an attachment. It uses IAttachment to get a transaction (single database) or IDtc/IDtcStart for a multi-database transaction. On 05/12/2019 10:31, Jiří Činčura wrote: 3. A transaction (ITransaction) is disposed of with a "release" 4. An attachment (IAttachment) is disposed of with a "detach". Interesting. I'm getting the transaction and attachment from the IExternalContext which si probaby slightly different. And now I'm wondering whether I should release those two at all. I suppose only core guys can tell me for sure. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] IResultSet and IStatement releasing
Jiři,
I've already had to go through the pain of getting the IResultSet and
IStatement interface to work when developing the Firebird Pascal API
that is distributed with IBX. The strategy that I found that works is:
1. A cursor (IResultSet) is best disposed of with "close" unless the
transaction is no longer active when "release" will do.
2. A statement (IStatement) is disposed of with a "release".
3. A transaction (ITransaction) is disposed of with a "release"
4. An attachment (IAttachment) is disposed of with a "detach".
In all cases, you should null the pointer to the interface after
disposing of it.
The Firebird Pascal API is object oriented and more than one copy of an
object can reference the same underlying Firebird interface, hence it
always uses reference counter interfaces to dispose of objects.
Regards
Tony Whyman
MWA
On 05/12/2019 09:01, Jiří Činčura wrote:
Maybe better to show some code (simplified). Expecting the free and release to
have separate responsibilities I wrote initially (the variables with underscore
are class fields). The Execute is called at some point and Free is called
either after the IExternalFunctionImpl finished executing or in
IExternalResultSetImpl::dispose (because that's the only place I'm aware of I
know no more fetching will occur and the fetching might end up in the middle
not reading all the results (otherwise I would know it because fetchNext would
return != IStatus::RESULT_OK).
void Execute(const char* stmt)
{
auto status = ThrowStatusWrapper(_context->getMaster()->getStatus());
_attachment = _context->getAttachment();
_transaction = _context->getTransaction();
_statement = _attachment->prepare(, _transaction, 0, stmt,
SQL_DIALECT_CURRENT, 0);
_cursor = _statement->openCursor(, _transaction, nullptr,
nullptr, outMetadata, 0);
_msg = new unsigned char[outMetadata->getMessageLength()];
// other method does the fetchNext
}
void Free()
{
_cursor->close();
_statement->free();
delete _msg;
_msg = nullptr;
_cursor->release();
_cursor = nullptr;
_statement->release();
_statement = nullptr;
_transaction->release();
_transaction = nullptr;
_attachment->release();
_attachment = nullptr;
}
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Query statistics
There's also the isc_dsql_sql_info() API call (see chapter 6 of the API Guide). This gives statistics on a per SQL statement basis, and allows you to get the number of inserts, updates, deletes, etc. performed by the statement. Note that in the Firebird 3 API this is implemented as the IStatement.getInfo API call. On 01/08/2019 10:14, Vlad Khorsun wrote: 31.07.2019 15:06, Jiří Činčura wrote: Hi *, is it possible to get statistics about the executed query similar to what isc_info_req_insert_count etc. allows? I'm mostly interested in indexed/non-indexed read counts, but whetever is available I'll include in the implementation. These stats counters are available at the attachment level. See isc_database_info() and tags below: isc_info_read_seq_count, isc_info_read_idx_count, isc_info_insert_count, isc_info_update_count, isc_info_delete_count, isc_info_backout_count, isc_info_purge_count, isc_info_expunge_count Note, counters above are accumulated since attachment start, thus one have to query stats two times (before and after statement execution) and calculate difference. More details is available at API Guide, see "Database operation counts" at chapter 4 "WORKING WITH DATABASES". Hope it helps, Vlad Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] isc_database_info and current database user
I am just wondering if the requested feature is not already present. A quick test with isc_database_info, with SYSDBA and a normal user logged in shows: - SYSDBA connection: only SYSDBA appears to be returned. - User connection: only user name is returned. From the security viewpoint, only a System Administrator should be able to see a complete list of user names - which is true when using the MON$ATTACHMENTS pseudo table. It looks like isc_database_info could be broken in that it appears not to allow a SYSDBA to see this list. However, I would argue that MON$ATTACHMENTS should always be preferred hence this is not an issue. Otherwise, this whole thread seems to be an argument about whether it is easier to use isc_database_info (with the consequential difficulty of unpacking the information returned) or using the normal database API. If you don't like using low level APIs, then maybe a high level one like IBPP (C++), the PHP interbase API or IBX should be used as their whole point is to make common operations easy for the programmer and each, in their own way, makes "Select Current_User from RDB$DATABASE" a very simple operation to perform. On 27/02/2019 09:30, Kovalenko Dmitry wrote: Hello, Could anybody tell me – how I can get name of current user through isc_database_info API function? I see isc_info_user_names. But I want to get the only one name. Thanks, Dmitry Kovalenko. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Authentication plugin mismatch not clearly reported to client
On 25/06/18 11:17, Dimitry Sibiryakov wrote: 25.06.2018 11:29, Tony Whyman wrote: Even if it were still computationally infeasible to break Srp today, it is probably that in the next few years it will be totally broken. You missed my words "non-theoretical". There is nothing theoretical about brute force attacks. They always work, the only issue how long they take. Security is an "arms race" and you to keep upping your game as you know that the attackers are only just behind you. Its a similar story with RSA key lengths. Once 1024 bits was impregnable. Now at least 2048 is recommended - some would say 4096 bits is what you need. In all cases, the trick is to keep an eye on how quickly brute force attacks are developing and to upgrade not when the attack becomes a "non-theoretical" attack - but before. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Authentication plugin mismatch not clearly reported to client
On 25/06/18 10:14, Alex Peshkoff via Firebird-devel wrote: On 25.06.2018 12:02, Dimitry Sibiryakov wrote: 25.06.2018 10:35, Alex Peshkoff via Firebird-devel wrote: Afraid you are wrong here. It helps an attacker to detect what plugin is actually used by server (for example - srp or srp256) and use that info to attack particular plugin later. Does srp have non-theoretical vulnerability? Bruteforce passwords over the wire. We are still missing any passwords regulation (like min.length, UP/low letters, etc.) i.e. people can use passwords like 'pass' and such things can be bruteforced. See also http://tracker.firebirdsql.org/browse/CORE-5789 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Authentication plugin mismatch not clearly reported to client
On 25/06/18 10:02, Dimitry Sibiryakov wrote: 25.06.2018 10:35, Alex Peshkoff via Firebird-devel wrote: Afraid you are wrong here. It helps an attacker to detect what plugin is actually used by server (for example - srp or srp256) and use that info to attack particular plugin later. Does srp have non-theoretical vulnerability? The problem with Srp is that it uses SHA-1 to generate the Client Proof. The Client Proof is itself a hash of several items including the shared secret. If it is possible to mount a brute force attack that allows the original message to be recovered from the SHA-1 hash then the shared secret is revealed. Given that this secret is also used for Wire Encryption then it becomes possible to an attacker to eavesdrop on on all data exchanged during the session, and which may include a database encryption key, if this is also exchanged. NIST has recommended against the use of SHA-1 in such situations because of the many reports from researchers of SHA-1 collision predictability and which may be used to speed up brute force attacks. Even if it were still computationally infeasible to break Srp today, it is probably that in the next few years it will be totally broken. The recommendation is thus to move to (e.g.) SHA-256 which is believed to be much less vulnerable to brute force attacks and the proposed patch is imply implementing the NIST guidance. It is also perhaps worth recalling that many organisations will have policies in place to avoid the use of products that do not comply with NIST recommendations. Hence, even if the vulnerability is still theoretical, it does help the acceptability of Firebird if it only offers SHA-1 based Srp. The purpose of the patch is to offer users the option of a better Client Proof and to avoid Firebird being rejected simply for reasons of organisational policy. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] exception handling in firebird.pas
Perhaps worth noting that the version of Firebird.pas distributed with IBX for Lazarus is patched with many changes. See https://svn.mwasoftware.co.uk/viewvc/public/ibx/trunk/fbintf/client/3.0/firebird/Firebird.pas?revision=209=markup This is: 1. To comment out all exception handling. This is to allow the IBX code to check for and handle the exceptions itself. In some cases, it allows for the exception to be returned as an error code rather than as an exception. This avoids the library user complaining that there is a bug in IBX when the exception appears in the IDE - this is in cases where IBX automatically handles the error code returned and does so otherwise silently. 2. For Unit structure reasons. Many of the Firebird constants are commented out and moved to a separate include file - actually restoring the original 'C' header file structure. This allows them to be referenced from the user interface without having to include the "firebird.pas" unit. This is particularly useful in allowing for a common interface for both the Firebird 3 API and the legacy API. It also avoids type name clashes. There are common type names (for different types) between IBX and the Firebird.pas unit (e.g. IStatement). Firebird.pas also does not follow type naming conventions for Delphi/FPC. IStatement should be an interface type, while it is a class in firebird.pas. In IBX, IStatement is the type name in IBX for a Pascal Interface to an SQL Statement. Whenever a Firebird.pas type is used in IBX, it is always scoped by prefixing it with "firebird." and the name of the "firebird" unit is carefully placed in the correct order in the calling unit's "uses" clause. This avoids any type name clashes within the IBX library itself, If IBX users had to do the same thing then novice users would find IBX difficult to use because of type name clashes. Hence, the need to avoid IBX users from having to include the "firebird.pas" unit in any of their unit's "uses" clauses. 3. The type name ISC_QUAD_Ptr is replaced with PISC_QUAD. This is again for compatibility reasons with the legacy API interface. On 03/05/18 08:29, Alex Peshkoff via Firebird-devel wrote: On 04/30/18 11:20, Mark Rotteveel wrote: On 30-4-2018 09:52, fbbt wrote: 3. Maybe we need to put a sample of firebird.pas at github. It will allow other developers to patch it in more productive way than now. As I understand it, firebird.pas is generated by Cloop, so allowing people to patch it would be against the intent. The problem would need to be fixed in the code generator: https://github.com/asfernandes/cloop (if I'm not mistaken, maybe Adriano can better address this). Useful suggestions (including patches) from other developers are anyway welcome - quite possible that some of them can be added to cloop. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-5789) Reducing the Risk of Brute Force attacks used to reveal Firebird User Passwords
Reducing the Risk of Brute Force attacks used to reveal Firebird User Passwords --- Key: CORE-5789 URL: http://tracker.firebirdsql.org/browse/CORE-5789 Project: Firebird Core Issue Type: Improvement Components: Security Affects Versions: 3.0.3 Environment: All Reporter: Tony Whyman User authentication mechanisms such as SRP are already resistant to brute force attacks where an attacker cycles through many different passwords in an attempt to login to a user account. However, a brute force attack will always succeed given a long enough period in which to sustain the attack. It is thus important that attackers are given limited windows of opportunity in which to conduct a brute force attack. At present, it appears that Firebird allows a user an unlimited number of failed login attempts and without reporting a possible brute force attack. The only mitigation to brute force attacks appears to be an eight second delay inserted after every fourth failed login (see src/remote/server/server.cpp in the Firebird source tree). This still allows an attacker to cycle through 60/8 * 4 * 60 * 24 = 43200 failed logins per day. That is after only four days or so an attacker could cycle through all 171,476 words in the Oxford English Dictionary. Given that Firebird does not enforce any rules for password selection, the tendency for users to select common passwords, the availability of dictionaries of well used passwords and the lack of any alert sent to a DBA to warn about an attack, it probably will not take many days to break a Firebird user's password and without anyone being aware that this has been done. There is a need both to log failed login attempts, so that a System Administrator is aware of a possible attack, and to rate limit login attempts in order to frustrate the attacker. http://tracker.firebirdsql.org/browse/CORE-5786 has reported the need to record failed login attempts in a suitable log file. Additionally, Firebird should also implement some form of advanced rate limiting to restrict the rate of failed login attempts for a given user. This might be by: * Inserting a delay time between receiving a failed login request and reporting the failure to the client. * An exponential increase in the delay time for each successive failed login between receiving a failed login request for a given user and reporting the failure to the client. * Rate limiting the total number of failed login attempts for each user from a given remote system during a given period to a configurable number. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-5788) Proposed Security Patch: Replacement of use of SHA-1 in the SRP Client Proof with SHA-256
Proposed Security Patch: Replacement of use of SHA-1 in the SRP Client Proof with SHA-256 - Key: CORE-5788 URL: http://tracker.firebirdsql.org/browse/CORE-5788 Project: Firebird Core Issue Type: Bug Components: Security Affects Versions: 3.0.3 Environment: All Reporter: Tony Whyman Attachments: srp_sha256.patch This proposed patch results from a security review of the Firebird SRP-6a implementation taking into account current NIST guidance on the use of SHA-1 - see NIST Special Publication 800-131A, Revision 1, Transitions: Recommendation for Transitioning the Use of Cryptographic Algorithms and Key Lengths (http://dx.doi.org/10.6028/NIST.SP.800-131Ar1) chapter 9. This guidance disallows the general use of SHA-1 for "Digital Signature Generation" whilst permitting continued use for "Digital Signature Verification". Review of the Firebird SRP implementation appears to indicate that most uses of SHA-1 continue to be permitted under NIST guidance except for its use in generating the client proof. The SRP client proof may be characterised as a "Poor Man's Digital Signature" in that it provides a two party proof of identity rather than the third party proof normally expected from a Digital Signature i.e. it is not a non-repudiable proof. Nevertheless, it is believed that generation of the client proof falls under the heading of "Digital Signature Generation" when considering the NIST Guidance. Continued use of SHA-1 in order to generate the client proof appears to risk leakage of the shared session key used to encrypt "over-the-wire" encryption and which hence also provides peer entity authentication during the lifetime of the connection. This may result in an attacker being able to monitor confidential communication either during the connection or at some later date and this could include leakage of an encryption key used to encrypt the user database, if this is passed from client to server during the connection. Such an attack is viable if weaknesses in SHA-1 can be exploited to allow a brute force attack on the client proof to be computationally feasible. All parts of the message on which the client proof is based may be known to an attacker with the exception of the shared session key and such an attack would concentrate on revealing this key. If it were possible to reveal the shared session key in real time then additionally a man-in-the-middle attack would be feasible. The severity of this issue is viewed as Important but not Critical. This is because (a) users that comply with NIST Guidance as a matter of policy may feel unable to use Firebird/SRP and hence choose or migrate to a different database, and (b) users that rely on SRP/over the wire encryption to protect confidential communication have a long term risk that the confidentiality of their data may be compromised. The attack may also be mitigated through the use of other procedures to protect communications (e.g. a secure VPN). The patch adds a new directory to the source code tree (src/common/sha2) containing an implementation of the SHA-2 family of message digests derived from the implementation published by Olivier Gay <olivier@a3.epfl.ch> (see https://github.com/ouah/sha2). This has been adapted for Firebird as a set of classes that follow the model of the existing Firebird::Sha1 class. Classes are provided for SHA-224, SHA-256, SHA-384 and SHA-512. A SHA-2 compliancy confidence test is also included. The SRP RemotePassword class is modified to additionally include a method for generating a client proof using SHA-256 as the message hash. The SRP client class is modified to use only SHA-256 for generating the client proof. The SRP server class is modified to use either SHA-1 or SHA-256 for verifying the client proof, with the verification method depending on the length of the client proof. This is believed to be compliant with NIST Guidance for legacy use of SHA-1 and permits backwards compatibility with older clients. The patch also modifies the makefiles for posix builds in order to include the SHA-2 classes in the "common" library and has been tested on Linux. It may be necessary to modify the build procedures for other platforms in order to use the patch. It is proposed that this patch is expedited into the next step release of Firebird 3 and which should be made available as soon as possible. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[Firebird-devel] [FB-Tracker] Created: (CORE-5787) Lack of Predictability in SET TRUSTED ROLE
Lack of Predictability in SET TRUSTED ROLE -- Key: CORE-5787 URL: http://tracker.firebirdsql.org/browse/CORE-5787 Project: Firebird Core Issue Type: Improvement Components: Security Affects Versions: 3.0.3 Environment: All Reporter: Tony Whyman Priority: Minor The "SET TRUSTED ROLE" SQL statement was added in Firebird 3 and is described in the release notes as: "The idea of a separate SET TRUSTED ROLE command is that, when the trusted user attaches to a database with-out providing any role info, SET TRUSTED ROLE makes a trusted role (if one exists) the CURRENT_ROLE without any additional activity, such as setting it in the DPB. A trusted role is not a specific type of role but may be any role that was created using CREATE ROLE, or a predefined system role such as RDB$ADMIN. It becomes a trusted role for an attachment when the security objects mapping subsystem finds a match between the authentication result passed from the plug-in and a local or global mapping for the current database. The role may be one that is not even granted explicitly to that trusted user." This lack of predictability (e.g. the last statement in the above quote) may be of concern to Security Administrators which want to ensure that all access rights are explicit and deterministic. In order to mitigate such concerns it is proposed that a configuration option is provided to disable this feature either globally or on per database basis. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-5765) Missing directives in Firebird.pas
Missing directives in Firebird.pas
--
Key: CORE-5765
URL: http://tracker.firebirdsql.org/browse/CORE-5765
Project: Firebird Core
Issue Type: Bug
Components: API / Client Library
Affects Versions: 3.0.3
Environment: Free pascal Compiler
Reporter: Tony Whyman
Priority: Minor
In order to compile and use with FPC, the Firebird.pas file requires the
following directives to be placed at the top of the file:
{$IFDEF FPC}
{$mode delphi}
{$OBJECTCHECKS OFF}
{$ENDIF}
If mode delphi is not specified then it will not compile unless this mode is
specified on the command line.
If Objectchecks off is not specified then if the using program is compiled with
the -CR debugging switch (Verify object method call validity) then a run-time
error is reported whenever an attempt is made to call a Firebird 3 API method.
The Firebird.pas file in the source code distribution needs to be updated to
include the above directives.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://tracker.firebirdsql.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] IBX2 is now available with full support for the FIrebird 3 API
For those interested in using Firebird from Pascal/Lazarus, I've copied the following announcement to this list. Tony Whyman MWA Software Announcement follows: MWA Software is pleased to announce the release of IBX2 for Lazarus. This is formally a pre-release. Although the software has been put through more extensive testing that any previous release of IBX for Lazarus, there are nevertheless considerable changes to the codebase and a period of in service experience is needed before IBX2 should be considered for production use. User reports, both positive and negative, are encouraged. IBX2 is available for download from http://www.mwasoftware.co.uk/ibx2 The IBX2 headlines are: * Full support for both the Firebird 3 API and the legacy Firebird API. * Support for Firebird Arrays including a new Data Aware control derived from a TCustomStringGrid. * Reduced RAM footprint. * Support for embedded SQL statement execution. * Extensive testing has been performed with emphasis on avoiding memory leaks and to ensure that the performance is at least as good as previous versions. * Further bug fixes (see changelog). IBX2 requires a minimum of FPC 3.0.0 and Lazarus 1.6.0. The original IBX code has been split into two packages: the main IBX package which provides the non-visual components, and a new /fbintf/package (see separate announcement). The /fbintf/package provides a common interface for the Firebird 3 API and the legacy Firebird API and is the means by which embedded SQL statement execution is supported. The /fbintf/package may also be used on its own and has the potential to provide a common Firebird API to all FPC database applications. IBX2 includes much improved documentation. There is a 108 page user guide for IBX itself and an 89 page user guide to the Firebird Pascal API provide by the /fbintf/package. There are also 15 example programs included in the package. IBX2 is intended to be backwards compatible and most user applications should re-compile with no issues. However, there are differences and the user guide includes a section on upgrading to IBX2. -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/xeonphiFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reliably Determining whether Engine12 is available as a Provider
On 22/11/16 16:56, Alex Peshkoff wrote: > On 11/22/16 16:42, Tony Whyman wrote: >> I am trying to find a way of determining whether Engine12 is available >> as a provider when using the FIrebird 3 API. >> >> Enumerating the list of Providers in firebird.conf tells you whether it >> has been configured but because the list is interpreted permissively, >> all this tells you is that it might be available and not will be available. >> >> IPluginManager.getPlugins looks hopeful, but calling >> >> getPlugins(status,TYPE_PROVIDER,"Engine12",null) >> >> returns a nil pointer, while >> >> getPlugins(status,TYPE_PROVIDER,"engine12",null) >> >> returns an IPluginSet with an empty getName and repeated calls to >> IPluginSet.next get you nowhere but an unending loop. >> >> Any ideas? > Do you have default for Providers in .conf files? Yes. But as I understand it if the engine12 library file is not installed then even though I have the default list of providers, that doesn't matter as the missing engine12 is silently ignored. > > > -- > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Reliably Determining whether Engine12 is available as a Provider
I am trying to find a way of determining whether Engine12 is available as a provider when using the FIrebird 3 API. Enumerating the list of Providers in firebird.conf tells you whether it has been configured but because the list is interpreted permissively, all this tells you is that it might be available and not will be available. IPluginManager.getPlugins looks hopeful, but calling getPlugins(status,TYPE_PROVIDER,"Engine12",null) returns a nil pointer, while getPlugins(status,TYPE_PROVIDER,"engine12",null) returns an IPluginSet with an empty getName and repeated calls to IPluginSet.next get you nowhere but an unending loop. Any ideas? Tony Whyman MWA -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 3 Debian Packages
Of course what this does seem to imply is that you can't scan the list of providers in the configuration file (e.g. using the Firebird API) to reliably determine which providers are available. On 21/11/16 16:01, Alex Peshkoff wrote: Missing providers are ignored. >Would it be better if the firebird.conf file >correctly reflected the installed plugins? A little better but not a big difference. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Firebird 3 Debian Packages
Just scanning through the Firebrid 3 Debian packages for Debian "stretch" and I noticed that: 1. if you install the libfbclient2 package only, it will call in the dependency for firebird3.0-common which includes the "firebird.conf" file. 2. This is the "as distributed" firebird.conf and include "Engine12" in the list of providers. However, libengine12 is not installed. You need also to install the firebird3.0-server-core to get this installed. Is this a problem? Would it be better if the firebird.conf file correctly reflected the installed plugins? Tony Whyman MWA -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Compiling Firebird.pas on Free Pascal
I put it at the beginning
On 05/09/16 10:22, Alex Peshkoff wrote:
> On 09/04/2016 06:26 PM, Tony Whyman wrote:
>> {$ifdef fpc} {$mode delphi} {$endif}
> Yes -Mdelphi is required switch.
>
> And where should be that line added? In the most beginning of firebird.pas?
>
>
>
> --
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
--
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Compiling Firebird.pas on Free Pascal
Martin,
I had the same problem. You have to add
{$ifdef fpc} {$mode delphi} {$endif}
in order to compile with FPC.
Tony Whyman
MWA
On 04/09/16 14:29, Martin Schreiber wrote:
> On Sunday 04 September 2016 10:34:05 Martin Schreiber wrote:
>> Hi,
>> Currently I am implementing Firebird 3.0 support in MSEide+MSEgui.
>>
>> Should Firebird.pas compile on FPC 3.0.1? I get
>> "
>> Free Pascal Compiler version 3.0.1 [2016/08/08] for i386
>> Copyright (c) 1993-2015 by Florian Klaempfl and others
>> Target OS: Linux for i386
>> [...]
>> firebird.pas(559,20) Error: Duplicate identifier "VERSION"
> For the record: It compiles with
> "
> {$ifdef fpc} {$mode delphi} {$endif}
> "
> Martin
>
> --
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
--
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Slice Description Language
I'm experimenting with the new Firebird 3 API and have come up against the buffers with array handling. The IAttachment interface has getSlice and putSlice methods - so far so good - but instead of the well known array descriptor these calls want data in the format of SDL (Slice Description Language) and parameters, which seems to be some sort of internal data structure. I have traced the code down to jrd/blb.cpp and common/sdl.cpp but without documentation it is really difficult to work out what is going on and how to create the SDL block needed for the getSlice and putSlice calls. Is there any documentation or examples on using this interface? Tony Whyman MWA -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] optional SET TERM
Yes, the number of times I have caught out by that little oddity. On 16/10/15 14:29, Alex Peshkoff wrote: > Which must be present in the end of struct or class The underlying point is that an sql script parser doesn't have to do a complete analysis but does have to break up the text into blocks of text that should be passed to an SQL Prepare statement. A simple rule could be to use a semi-colon as the block separator - except that that is too simple. At the very least, a simple parser has to ignore semi-colons in quoted text and comments and ignoring semi-colons inside BEGIN..END blocks is not that difficult either. Indeed, exiting the outermost BEGIN..END block is a good enough rule for passing the current text block to a prepare statement. Requiring a following semi-colon or a '^' is just syntactic candy. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] optional SET TERM
Except that there is no more reason for a separator after the final PSQL END statement than there is in 'C' for a semi-colon after a closing brace. On 16/10/15 14:01, Ivan Přenosil wrote: > There are actually two separators/terminators - one to be used in scripts > to separate commands, one to be used inside PSQL commands. > The whole problem is caused by the fact that somebody chose > the same character for both. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] optional SET TERM
Hmmm, I'm not even sure that "complicates" is even the right word here - counting BEGIN/END block nesting is not the most complex of programming problems. Several years ago I wrote my own SQL statement parser (in Pascal) to automate database upgrades without depending on ISQL being available. My recollection is that it was more complicated adding ISQL set term compatibility (i.e. having to parse a set term statement) than it was to count begin/end block nesting and ignore semi-colons except in the outer block. Anyway, the parser worked regardless of whether set term was used. Tony Whyman MWA On 15/10/15 16:34, Ann Harrison wrote: >> On Oct 15, 2015, at 9:59 AM, Dimitry Sibiryakov <s...@ibphoenix.com> wrote: >> >> 15.10.2015 15:51, marius adrian popa wrote: >>> In InterBase 7 is changed so procedure and trigger programming language to >>> no longer >>> require the use of SET TERM >>IMHO, this is unnecessary complication of isql's parser. I'd prefer to >> follow KISS concept. > Respectfully disagree. Yes, it complicates isql, but it makes the user's > life easier. One bit of complication in isql, thousands of simpler to > create triggers and procedures. > > Cheers, > > Ann > -- > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Roadmap for a Distributed Firebird
Just out of interest, is OCFS2/DRBD any use as a page/lock server? On 23/03/15 13:01, James Starkey wrote: The major new piece is a separate component, the page server. By phase 5, it will be necessary to integrate page change and lock manager traffic, so perhaps a better name would be page/lock server. Code historians may find existing references to an ancient page/lock server, circa 1987, which I never competed. -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
On 15/08/14 08:48, Dimitry Sibiryakov wrote:
Integers are ok, pointers are ok, interfaces are in trouble
This should probably be written in stone somewhere.
Even simple external calls such as fb_get_master_interface are difficult
enough. You have carefully define each data type used and ensure that
the calling conventions are compatible. Even cdecl starts being a
problem as soon as you get away from simple types - as you have found.
When it comes to an OO interface you also need to align the vtable
navigation and how the instance variable ('this' in c++ ) is passed -
and if the two compilers have different views on this area then there is
not much that can be done.
--
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
I'm not quite sure what you think is wrong with this code (from the fpc viewpoint): # [31] WriteLn(master.getVersion()); callfpc_get_output movq%rax,%rbx movqU_P$CREATE_MASTER,%rdi movqU_P$CREATE_MASTER,%rax movq(%rax),%rax call*(%rax) movl%eax,%edx movslq%edx,%rdx movq%rbx,%rsi movl$0,%edi callfpc_write_text_sint callFPC_IOCHECK movq%rbx,%rdi callfpc_writeln_end callFPC_IOCHECK As far as I can make out, it is loading the instance variable (Master) into %rdi movqU_P$CREATE_MASTER,%rdi It then assumes a simple vtable and loads and calls the first pointer in the table: movqU_P$CREATE_MASTER,%rax movq(%rax),%rax call*(%rax) On return, the result is assumed to be in $eax (standard cdecl for a 32-bit integer return), and then, after a sign extend to 64-bit passes it on to the writeln: movl%eax,%edx movslq%edx,%rdx movq%rbx,%rsi movl$0,%edi callfpc_write_text_sint It would be interesting to see what the c++ compiler expected, but my guess is that it does not use %rdi for the instance variable. On 14/08/14 12:28, Dimitry Sibiryakov wrote: 14.08.2014 11:34, Alex Peshkoff wrote: HP C/aC++ Version A.06.26 Alpha version?.. PS: The problem with FPC which started whole this topic seems to be a compiler's bug which destroy value returning by external function. Testcase is attached. I registered ticket in their bugtracker: http://bugs.freepascal.org/view.php?id=26593. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
Just came across this on Wikipedia (hence needs to be checked) If this is how gcc ++ works then it is clear why there is a mis-match with fpc (instance variable in %rdi) and IMaster.GetVersion (instance variable expected on stack with caller cleanup). /thiscall/ // /This calling convention is used for calling C++ non-static member functions. There are two primary versions of //*thiscall*//used depending on the compiler and whether or not the function uses variable arguments./ // /For the GCC compiler, //*thiscall*//is almost identical to //*cdecl*//: The caller cleans the stack, and the parameters are passed in right-to-left order. The difference is the addition of the //*this* pointer http://en.wikipedia.org/wiki/This_%28computer_programming%29//, which is pushed onto the stack last, as if it were the first parameter in the function prototype./ On 14/08/14 13:08, Tony Whyman wrote: It would be interesting to see what the c++ compiler expected, but my guess is that it does not use %rdi for the instance variable -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
Apologies, I should have read your bug report before replying as I didn't realise your problem was with the flat call to fb_get_master_interface. I just assumed that you would be getting a problem with the call to GetVersion. I have put your Pascal example into my test system and got an Access Violation error when IMaster.GetVersion is called. Looking at the code, you may be a bit hopeful in giving the return type of GetMaster as IMaster - FPC may assume too much from this. I changed it to: function GetMaster: Pointer; cdecl; external 'libfbclient.so' name 'fb_get_master_interface'; to get back a simple pointer and then coerced the result to IMaster as master := IMaster(GetMaster()); The program now does complete without an error, but returns 15 as the version. My reading of the Firebird code is that the correct result is 4. So still something not right - probably the way the instance variable is passed. Using the lazarus disassember, the call to fb_get_master_interface now looks like this i.e. much more like a cdecl call returning a simple type. 004003D4 48c7042590ba650078563412 movq $0x12345678,0x65ba90 project1.lpr:24 master := IMaster(GetMaster()); 004003E0 e83bff callq 0x400320 fb_get_master_interface@plt 004003E5 48a390ba6500 movabs %rax,0x65ba90 Interestingly, if I change the coercion to master := TObject(GetMaster()) as IMaster; It gives me an access violation on the call to GetMaster - or at least the internal call that validates the interface - so there really is something different between the two structures. On 14/08/14 14:56, Dimitry Sibiryakov wrote: 14.08.2014 14:08, Tony Whyman wrote: I'm not quite sure what you think is wrong with this code (from the fpc viewpoint): That code is fine. This isn't: callfb_get_master_interface movq-104(%rbp),%rax movq%rax,U_P$CREATE_MASTER As you can see, value returned by function in %rax is overwritten before it get assigned to the variable. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
I was using the Alpha 2 source On 14/08/14 15:23, Adriano dos Santos Fernandes wrote: On 14/08/2014 11:04, Tony Whyman wrote: master := IMaster(GetMaster()); The program now does complete without an error, but returns 15 as the version. My reading of the Firebird code is that the correct result is 4. So still something not right - probably the way the instance variable is passed. Version from IMaster from trunk should be 17. Adriano -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
Mark, A good post as this sums up the issue. I only hope that the core team are listening as what happens to a project when the developers ignore the users? On 12/08/14 09:13, Mark Rotteveel wrote: The main point of an API is to allow developers to*use* Firebird, so please do not alienate those users as their concerns are valid and should be taken into account. And if people already using Firebird voice those concerns, what do you think happens if people evaluate Firebird for use? If the choice is between a legacy API that doesn't expose all (modern) features of Firebird or a C++ API that is only really usable from C++ or requires catering to the demands of some weird calling mechanism (note: I am exaggerating), then they'd probably take their business elsewhere. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
