Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 5/20/21 5:32 PM, Adriano dos Santos Fernandes wrote:
Hi!
Currently max. CHAR length is limited to 32767 and max. VARCHAR to 32765.
I didn't found any blocker prevent raising that limits to 65535 and
65533 respectively.
Even not going into details agraid I see one serious problem - ISC API
compatibility.
Look at XSQLVAR:
typedef struct
{
ISC_SHORT sqltype; /* datatype of field */
ISC_SHORT sqlscale; /* scale factor */
ISC_SHORT sqlsubtype; /* datatype subtype - currently
BLOBs only */
ISC_SHORT sqllen; /* length of data area */
sqllen is 16 bit, and it's signed. I.e. people will see negative lengths
if variables in sqlda. I'm afraid at least 99% of software working with
sqlda does not check for negative length, and receiving it will cause
undefined behavior.
Any idea how to avoid that?
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
21.05.2021 09:38, Alex Peshkoff via Firebird-devel wrote: Any idea how to avoid that? Redeclare it as unsigned right now. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 21/05/2021 08:33, Dimitry Sibiryakov wrote: > 21.05.2021 09:38, Alex Peshkoff via Firebird-devel wrote: >> Any idea how to avoid that? > > Redeclare it as unsigned right now. > Yes. But would be very bad to see not recompiled programs just crashing (or in infinite loop) if greater length is returned. So I think we must also create SQLDA_VERSION2 and raise error with length > 32767 with SQLDA_VERSION1. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Duplicated boost in Windows v4 RC1 kits
Hi! In v4 RC1 zip kits for Windows, there is duplicated boost directory. One in include/firebird/impl/boost (correct) and another in include/firebird/boost (incorrect). In recent GitHub actions artifacts, I see it's not there. But I don't see anything explicit in the logs since RC1, so I'm not sure there is a difference in how the official build is done. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
21.05.2021 14:49, Adriano dos Santos Fernandes wrote: So I think we must also create SQLDA_VERSION2 and raise error with length > 32767 with SQLDA_VERSION1. Creating of a new SQLDA version must be considered much deeper than just sqllen. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 5/21/21 3:49 PM, Adriano dos Santos Fernandes wrote: On 21/05/2021 08:33, Dimitry Sibiryakov wrote: 21.05.2021 09:38, Alex Peshkoff via Firebird-devel wrote: Any idea how to avoid that? Redeclare it as unsigned right now. Yes. But would be very bad to see not recompiled programs just crashing (or in infinite loop) if greater length is returned. So I think we must also create SQLDA_VERSION2 and raise error with length > 32767 with SQLDA_VERSION1. Just to make sure - raise error on prepare or when actual length > 32767? And if we do change SQLDA_VERSION it's worth changing something else in it. For example - make all sizes uint32, add separate field for charset, may be something else? If we increase sizes - let me return to your initial email. But I want to raise them both to 65535 to allow read 64KB segments in RDB$BLOB_UTIL. 65535 is 64KB - 1 To support that, internals must be changed so that the 16-bit dsc_length variable for dtype_varying does not store +2 (varchar length field) added to dsc_length, as well not add +1 for dtype_cstring (not sure about cstrings - more later). We have one more unused value for size - zero, we do not support characters with length == 0. We may store logical data length - 1, this will make it possible to have exactly 64Kb fields, which (I believe) can be useful in some cases. dsc_length is currently the descriptor's physical length. With the change it will represent a logical length. To support that I added getPhysicalLength() method to struct dsc returning an unsigned 32-bit that sums +2/+1 for dtype_varying/dtype_cstring. During such cleanup may be make DSC fields private? What;s also worth taking into an account - what to do if user tries to include such huge field in an index. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
21.05.2021 16:51, Alex Peshkoff via Firebird-devel wrote: We may store logical data length - 1, this will make it possible to have exactly 64Kb fields, which (I believe) can be useful in some cases. Yep, mess with varying::vary_length. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Duplicated boost in Windows v4 RC1 kits
On Fri, 21 May 2021 11:12:27 -0300 Adriano dos Santos Fernandes wrote: > Hi! > > In v4 RC1 zip kits for Windows, there is duplicated boost directory. > > One in include/firebird/impl/boost (correct) and another in > include/firebird/boost (incorrect). > I think I've located the cause of the problem. I'll commit the fix once I'm sure. Paul -- Paul Reeves http://www.ibphoenix.com Supporting users of Firebird Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 2021-05-21 16:51, Alex Peshkoff via Firebird-devel wrote: We have one more unused value for size - zero, we do not support characters with length == 0. We may store logical data length - 1, this will make it possible to have exactly 64Kb fields, which (I believe) can be useful in some cases. Please don't apply that sort of hacks. It breaks existing implementations, and is extremely confusing. Mark Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 21/05/2021 11:51, Alex Peshkoff via Firebird-devel wrote: > On 5/21/21 3:49 PM, Adriano dos Santos Fernandes wrote: >> On 21/05/2021 08:33, Dimitry Sibiryakov wrote: >>> 21.05.2021 09:38, Alex Peshkoff via Firebird-devel wrote: Any idea how to avoid that? >>> Redeclare it as unsigned right now. >>> >> Yes. >> >> But would be very bad to see not recompiled programs just crashing (or >> in infinite loop) if greater length is returned. >> >> So I think we must also create SQLDA_VERSION2 and raise error with >> length > 32767 with SQLDA_VERSION1. > > Just to make sure - raise error on prepare or when actual length > 32767? > In isc_dsql_describe / isc_dsql_describe_bind. > And if we do change SQLDA_VERSION it's worth changing something else in > it. For example - make all sizes uint32, add separate field for charset, > may be something else? > My aim is to increase CHAR/VARCHAR lengths. I think there is no real benefit in improve the old API when the new one already exists and support that. > If we increase sizes - let me return to your initial email. > >> But I want to raise them both to 65535 to allow read 64KB segments in >> RDB$BLOB_UTIL. > > 65535 is 64KB - 1 > Sorry, I mean MAX_USHORT. >> >> To support that, internals must be changed so that the 16-bit dsc_length >> variable for dtype_varying does not store +2 (varchar length field) >> added to dsc_length, as well not add +1 for dtype_cstring (not sure >> about cstrings - more later). >> > > We have one more unused value for size - zero, we do not support > characters with length == 0. > We may store logical data length - 1, this will make it possible to have > exactly 64Kb fields, which (I believe) can be useful in some cases. > Not possible. >> dsc_length is currently the descriptor's physical length. With the >> change it will represent a logical length. To support that I added >> getPhysicalLength() method to struct dsc returning an unsigned 32-bit >> that sums +2/+1 for dtype_varying/dtype_cstring. > > During such cleanup may be make DSC fields private? > > What;s also worth taking into an account - what to do if user tries to > include such huge field in an index. > We already have this problem: SQL> create table t (c varchar(32765)); SQL> create index t_c on t (c); Statement failed, SQLSTATE = 42000 unsuccessful metadata update -CREATE INDEX T_C failed -key size too big for index T_C Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
> >> dsc_length is currently the descriptor's physical length. With the > >> change it will represent a logical length. To support that I added > >> getPhysicalLength() method to struct dsc returning an unsigned 32-bit > >> that sums +2/+1 for dtype_varying/dtype_cstring. > > > > During such cleanup may be make DSC fields private? > > > > What;s also worth taking into an account - what to do if user tries to > > include such huge field in an index. > > > > We already have this problem: > > SQL> create table t (c varchar(32765)); > SQL> create index t_c on t (c); > Statement failed, SQLSTATE = 42000 > unsuccessful metadata update > -CREATE INDEX T_C failed > -key size too big for index T_C Given that the max size of an index is not related to the max size of any datatype, how does increasing max varchar solve the that use case? Sean Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 5/21/21 9:06 PM, Mark Rotteveel wrote: On 2021-05-21 16:51, Alex Peshkoff via Firebird-devel wrote: We have one more unused value for size - zero, we do not support characters with length == 0. We may store logical data length - 1, this will make it possible to have exactly 64Kb fields, which (I believe) can be useful in some cases. Please don't apply that sort of hacks. It breaks existing implementations, and is extremely confusing. Mark, I've never suggested to use that size coding at places visible to users. Internal encoding used by engine can't break anuthing (at least directly). Though I'm not sure what did you mean under existing implementations, i.e. my answer can be incomplete. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 5/21/21 9:19 PM, Adriano dos Santos Fernandes wrote: And if we do change SQLDA_VERSION it's worth changing something else in it. For example - make all sizes uint32, add separate field for charset, may be something else? My aim is to increase CHAR/VARCHAR lengths. I think there is no real benefit in improve the old API when the new one already exists and support that. OK. Let's compare what do you obtain with needed for that changes. Incrementing XSQLDA version is rather serious change in old API which on the other hand you do not want to improve. I think that there should be very serious reasons for such change. Please explain what do we win in blob package with longer char/varchar. If we increase sizes - let me return to your initial email. But I want to raise them both to 65535 to allow read 64KB segments in RDB$BLOB_UTIL. 65535 is 64KB - 1 Sorry, I mean MAX_USHORT. Half-done solution at the first glance. Full 64Kb appears much more interesting. Agree - aligned block of memory which size is power of two is always prefered unit. I once again stop discussing details - let's begin with the beginning, why is increase of char/varchar important for BLOB utils. Is it alone interesting enough to change XSQLDA version? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
On 21/05/2021 17:53, Alex Peshkoff via Firebird-devel wrote: > On 5/21/21 9:19 PM, Adriano dos Santos Fernandes wrote: > >>> And if we do change SQLDA_VERSION it's worth changing something else in >>> it. For example - make all sizes uint32, add separate field for charset, >>> may be something else? >>> >> My aim is to increase CHAR/VARCHAR lengths. I think there is no real >> benefit in improve the old API when the new one already exists and >> support that. > > OK. Let's compare what do you obtain with needed for that changes. Allow reading full blob segments. > Incrementing XSQLDA version is rather serious change in old API which on > the other hand you do not want to improve. I think that there should be > very serious reasons for such change. It's easy and compatible change. > >> >>> If we increase sizes - let me return to your initial email. >>> But I want to raise them both to 65535 to allow read 64KB segments in RDB$BLOB_UTIL. >>> 65535 is 64KB - 1 >>> >> Sorry, I mean MAX_USHORT. > > Half-done solution at the first glance. Full 64Kb appears much more > interesting. Agree - aligned block of memory which size is power of two > is always prefered unit.> Varchar length (actual, not declared) could be between 0-MAX_USHORT (64K possible values). To support max. of 64K bytes, length field must be increased to 32 bits. If that is the solution, then a new descriptor/SQL type should be added. > I once again stop discussing details - let's begin with the beginning, > why is increase of char/varchar important for BLOB utils. Is it alone > interesting enough to change XSQLDA version? > The benefit for the old API is to support large strings. That's a benefit in itself, it has nothing directly to do with RDB$BLOB_UTIL. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Increasing CHAR/VARCHAR max. length to 64KB
21.05.2021 14:49, Adriano dos Santos Fernandes wrote: On 21/05/2021 08:33, Dimitry Sibiryakov wrote: 21.05.2021 09:38, Alex Peshkoff via Firebird-devel wrote: Any idea how to avoid that? Redeclare it as unsigned right now. Yes. But would be very bad to see not recompiled programs just crashing (or in infinite loop) if greater length is returned. For greater length to be returned there must be something that return it. If we keep old things unchanged, old programs will work with old databases without problems. Yes, at some point there will be requests to change returned value and parameters of the functions like LPAD but not right now so the application developers have time to adapt to changes. Alternatively a pseudo-type like SHORT_VARCHAR can be introduced to let BIND usage for the applications. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
