Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> In this case, it would be faster for both of us if you just sent some > (non-buildable) relevant pieces of code for now and I try to reproduce > with UDR or FB/Java. Sent in private email, with attachments. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 19/04/2017 07:53, Jiří Činčura wrote: >> It would be very good, specially if it works in Linux (.NET Core?). > .NET Core does not have a support for C++/CLI yet (if ever), so it's a > full .NET, thus Windows only. Would that work for you? > In this case, it would be faster for both of us if you just sent some (non-buildable) relevant pieces of code for now and I try to reproduce with UDR or FB/Java. Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
19.04.2017 13:43, Jiří Činčura wrote: >> Very strange. My test with UDR was with embedded. >> >> Even more strange because external procedures and remote/embedded layer >> has nothing related. > > I tried padding the CHAR with different character than 32 (space) - to > see whether it changes on plugin part as well -, but that didn't work. > It wasn't even passed to the plugin and ended somewhere sooner. > > What can I do to test it further? The value itself is created in .NET > provider the same way for server and for embedded. So either I'm > marshaling (that's where it differs from network obviously) it wrong > into embedded or there's a bug. > > I can create some sample for you, if you want. Then you can drop in your > own fbclient.dll and attach debugger or something like that. I can look at it if someone explains me what to look for exactly :) Regards, Vlad -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> It would be very good, specially if it works in Linux (.NET Core?). .NET Core does not have a support for C++/CLI yet (if ever), so it's a full .NET, thus Windows only. Would that work for you? -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 19/04/2017 07:43, Jiří Činčura wrote: > I can create some sample for you, if you want. Then you can drop in your > own fbclient.dll and attach debugger or something like that. > It would be very good, specially if it works in Linux (.NET Core?). Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> Very strange. My test with UDR was with embedded. > > Even more strange because external procedures and remote/embedded layer > has nothing related. I tried padding the CHAR with different character than 32 (space) - to see whether it changes on plugin part as well -, but that didn't work. It wasn't even passed to the plugin and ended somewhere sooner. What can I do to test it further? The value itself is created in .NET provider the same way for server and for embedded. So either I'm marshaling (that's where it differs from network obviously) it wrong into embedded or there's a bug. I can create some sample for you, if you want. Then you can drop in your own fbclient.dll and attach debugger or something like that. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 19/04/2017 03:33, Jiří Činčura wrote: >> I'm running it from Embedded now. I'll try tomorrow normal server. >> Shouldn't matter, but just in case. > Shouldn't, but does. Using server I get correct lengths, but using > embedded I always get 4*length (when using parameter, of course). > Very strange. My test with UDR was with embedded. Even more strange because external procedures and remote/embedded layer has nothing related. Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> you must decode input UTF8 buffer into string(probably unicode) and next > you have to trim trailing spaces. > For text, just put some non US characters into text. You can't trim all trailing spaces. You need to know the length. But even that is wrong, because I should get it as SQL_VARYING properly. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> I'm running it from Embedded now. I'll try tomorrow normal server. > Shouldn't matter, but just in case. Shouldn't, but does. Using server I get correct lengths, but using embedded I always get 4*length (when using parameter, of course). -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> This is some misuse or bug that requires a test case, because the > feature I described should work not only in external engines, but for > any query as the node driver shows in it test. I'm running it from Embedded now. I'll try tomorrow normal server. Shouldn't matter, but just in case. Is there anything I can do to test it further? > What about functions? I have only, at the moment, procedures working. Not sure I can do functions quickly enough. > I just tested the UDR example wait_event declaring the SQL with CHAR(31) > and the function that uses VARCHAR just received the relevant length. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 18/04/2017 15:35, Adriano dos Santos Fernandes wrote: > I just tested the UDR example wait_event declaring the SQL with CHAR(31) > and the function that uses VARCHAR just received the relevant length. I mean, it does not received 31 * 4, but received length 3 for string '123'. There is some controversy which I remember that I started a discussion if external engines should a conversion or replace the declared metadata type, and at least in this case, it replaced. Anyway, it's different problem than what you showed in the dump. Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
Hi Jiri,
you must decode input UTF8 buffer into string(probably unicode) and next
you have to trim trailing spaces.
For text, just put some non US characters into text.
Slavek
> The length from metadata is 80. The memory dump where the pointer points
> is:
> { 80, 0, 49, 50, 51, 52, 53, 54, 55, 56, 57, 48, 49, 50, 51, 52, 53, 54,
> 55, 56, 57, 48, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
> 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
> 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
> 32, 32, 32, 32, 32, 32, 32, 32, 32 }
>
> Which clearly shows the first two bytes are reporting 80 as well. Fair
> enough. Reading that buffer as UTF8 (or just looking at it, is basically
> plain US-ASCII, nothing fancy) you get
> "12345678901234567890"
> (I replaced the spaces with visible character). It's clearly correct
> string at the beginning and then 60(!) spaces. And that doesn't look
> correct to me at all.
>
> I think that's what Dimitry S. was pointing to.
>
> With all that, unless I'm doing something wrong, I'd need to do some
> trimming (or forgot about CHARs at all). The conversion to SQL_VARYING
> gets me only so far.
>
>> This change from SQL_TEXT->SQL_VARYING will not work only in some old
>> versions (maybe firsts 2.1.x or 2.0.x).
> Fine for me, as external engine needs 3.0 anyway.
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> Jiri, what you said (change SQL_TEXT to SQL_VARYING) and what I show you
> (fixMetadata function) just fix this problem.
I wish. See below.
Also I have to investigate the 32766 or 32767 limit Dmitry Y. mentioned.
I suppose he's not lying. :) That would make it suck even more.
> Change it and forget. You'll always read SQL_VARYING and have the actual
> length in the first two bytes.
This is what I'm getting from the server. The field is CHAR(20) UTF8.
The length from metadata is 80. The memory dump where the pointer points
is:
{ 80, 0, 49, 50, 51, 52, 53, 54, 55, 56, 57, 48, 49, 50, 51, 52, 53, 54,
55, 56, 57, 48, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,
32, 32, 32, 32, 32, 32, 32, 32, 32 }
Which clearly shows the first two bytes are reporting 80 as well. Fair
enough. Reading that buffer as UTF8 (or just looking at it, is basically
plain US-ASCII, nothing fancy) you get
"12345678901234567890"
(I replaced the spaces with visible character). It's clearly correct
string at the beginning and then 60(!) spaces. And that doesn't look
correct to me at all.
I think that's what Dimitry S. was pointing to.
With all that, unless I'm doing something wrong, I'd need to do some
trimming (or forgot about CHARs at all). The conversion to SQL_VARYING
gets me only so far.
> This change from SQL_TEXT->SQL_VARYING will not work only in some old
> versions (maybe firsts 2.1.x or 2.0.x).
Fine for me, as external engine needs 3.0 anyway.
--
Mgr. Jiří Činčura
https://www.tabsoverspaces.com/
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 18/04/2017 12:38, Jiří Činčura wrote: > Hmm. That explains a lot of issues I was trying to push through today > with CHARs. I.e. was getting 80 bytes buffer where was 20 characters > and 60 spaces, because the characters were all 1B. > > It's pretty sad, actually. > > So my best chance is probably to trim to length/4 and hope for the best. > > Jiri, what you said (change SQL_TEXT to SQL_VARYING) and what I show you (fixMetadata function) just fix this problem. Change it and forget. You'll always read SQL_VARYING and have the actual length in the first two bytes. The problem is not with CHAR data type, only its representation with SQL_TEXT. This change from SQL_TEXT->SQL_VARYING will not work only in some old versions (maybe firsts 2.1.x or 2.0.x). Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
It's always going to be spaces/0s. I don't see a way for otherwise. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ On Tue, Apr 18, 2017, at 17:43, Dimitry Sibiryakov wrote: > 18.04.2017 17:38, Jiří Činčura wrote: > > So my best chance is probably to trim to length/4 and hope for > > the best. > >Trim only spaces (zeros for OCTETS). > > > -- >WBR, SD. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdotFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
Hmm. That explains a lot of issues I was trying to push through today with CHARs. I.e. was getting 80 bytes buffer where was 20 characters and 60 spaces, because the characters were all 1B. It's pretty sad, actually. So my best chance is probably to trim to length/4 and hope for the best. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdotFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
18.04.2017 17:23, Mark Rotteveel wrote: > Real length in characters of a CHAR is length / max bytes per character, > so for UTF-8: length / 4. But if you count real character in this buffer, you can get up to 80 of them. -- WBR, SD. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 18-4-2017 17:20, Dimitry Sibiryakov wrote: > 18.04.2017 17:13, Jiří Činčura wrote: >> Hmm. Can you elaborate on that? > >Server doesn't determine real length of string in characters, so it just > pad string up > to 4*MaxChars bytes with spaces. That's why for CHAR(20) you always get 80 > bytes of data, > which can contain more that 20 characters if you count trailing spaces as > well. >And there is no place in API where real length in bytes can be provided > for CHAR() type. >Because of that handling of CHAR() is broken in many places for UTF-8 and > alike charsets. > Real length in characters of a CHAR is length / max bytes per character, so for UTF-8: length / 4. Mark -- Mark Rotteveel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
18.04.2017 17:13, Jiří Činčura wrote: > Hmm. Can you elaborate on that? Server doesn't determine real length of string in characters, so it just pad string up to 4*MaxChars bytes with spaces. That's why for CHAR(20) you always get 80 bytes of data, which can contain more that 20 characters if you count trailing spaces as well. And there is no place in API where real length in bytes can be provided for CHAR() type. Because of that handling of CHAR() is broken in many places for UTF-8 and alike charsets. -- WBR, SD. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
>CHAR and UTF-8 (generally, any variable character length charset) must >never be used > together. Hmm. Can you elaborate on that? Server allows that. And if developer creates such .NET UDR, I have to somehow handle it. Of course I can declare CHAR not supported at all, but that's kind of lame. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
18.04.2017 17:03, Jiří Činčura wrote: > So for CHAR(20) UTF8 CHAR and UTF-8 (generally, any variable character length charset) must never be used together. -- WBR, SD. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> See fixMetadata here in the Node.js driver. > > https://github.com/asfernandes/node-firebird-driver-native/blob/master/src/lib/fb-util.ts That's what I do exactly. But then I'm getting in length* full length of the field (as if it's CHAR) and not real length (as in VARCHAR). So for CHAR(20) UTF8, changed to SQL_VARYING, the length is 80. Which is kind of confusing. :) * You're reading it here: https://github.com/asfernandes/node-firebird-driver-native/blob/master/src/lib/fb-util.ts#L135 . -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
18.04.2017 16:52, Dmitry Yemanov wrote: > Generally it is, unless SQL_TEXT is of length 32766 or 32767. Isn't vary.vary_length declared as unsigned short? -- WBR, SD. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
18.04.2017 17:40, Jiří Činčura wrote: > > Also going from SQL_TEXT to SQL_VARYING should be possible, right? Generally it is, unless SQL_TEXT is of length 32766 or 32767. Dmitry -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
18.04.2017 16:40, Jiří Činčura wrote: > Isn't there a constant for i.e. > SQL_TIMESTAMP? Because it's always 8 bytes, but feels wrong to hardcode > it manually myself. sizeof(ISC_TIMESTAMP) -- WBR, SD. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> You should call setLength and setScale (if necessary) too. Ah, that's what I was missing. Isn't there a constant for i.e. SQL_TIMESTAMP? Because it's always 8 bytes, but feels wrong to hardcode it manually myself. Also going from SQL_TEXT to SQL_VARYING should be possible, right? How I'm going to handle the length here. I don't know the real length of the string, do I? -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 18/04/2017 11:27, Jiří Činčura wrote: >> Yes, sure. > Should changing the type be enough? Because I changed it from > SQL_TYPE_DATE to SQL_TIMESTAMP, but I'm still getting length == 4, while > the timestamp should be 8, isn't it? > You should call setLength and setScale (if necessary) too. Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
> Yes, sure. Should changing the type be enough? Because I changed it from SQL_TYPE_DATE to SQL_TIMESTAMP, but I'm still getting length == 4, while the timestamp should be 8, isn't it? -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Changing IRoutineMetadata in Plugin::makeProcedure
On 18/04/2017 11:15, Jiří Činčura wrote: > Hi *, > > in Plugin::makeProcedure I'm getting IRoutineMetadata and input and > output builder. Can I use these, somehow, to change what values the > server is going to send me later? I.e. calling getInputMetadata, > changing types (in my case currently from SQL_TYPE_DATE to > SQL_TIMESTAMP), then using builder->getMetadata and using this new > metadata later in my IExternalProcedureImpl? In other words, is then the > data passed in "void* msg" i.e. to Procedure::open going to be in new > format? > Yes, sure. Adriano -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
