Re: How do I send binary data to avatica?

Oh! That is easier. I had missed that detail that the b64-encoded data is being set in string_value (and not bytes_value). I was (incorrectly) thinking that we had to somehow differentiate between when bytes_values had b64 data and when it had "normal" data :) F21 wrote: I think the best way

Re: How do I send binary data to avatica?

Awesome work! This definitely sounds like a bug. It must be happening inadvertently in the translation. A JIRA issue for this would be great. It'll be tricky making this "just work" for older versions, but that'll be something to figure out when fixing it. - Josh F21 wrote: Hey Josh, I

Re: How do I send binary data to avatica?

Hey Josh, I was able to discover the source of the problem. I found your Github repo showing how to use the phoenix thin client here: https://github.com/joshelser/phoenix-queryserver-jdbc-client I hacked up CreateTables.java to create a table with a VARBINARY column and to read a file into

Re: How do I send binary data to avatica?

This is the test case I just wrote and it passes successfully (my knowledge of Java and its syntax is extremely limited, so hopefully it's testing the right things): @Test public void testInsertingFile() throws Exception { ConnectionSpec.getDatabaseLock().lock(); try (Connection

Re: How do I send binary data to avatica?

Francis, Sorry, I did not mean to imply that you were not doing your utmost to resolve this problem. There is no doubt that both sides are working diligently and in good faith. Regarding golden files. I totally agree that these are a useful pattern to follow. My “quidem” project[1] (used by

Re: How do I send binary data to avatica?

F21 wrote: Hey Josh and Julian, Thanks for your input regarding this. I also spent hours and hours over the last few days trying to get to the bottom of this, and it honestly wasn't the best use of my time either. I am sure being in different timezones also makes it much harder to this problem

Re: How do I send binary data to avatica?

+1 on some kind of test framework I can tell that Josh has put in a lot of effort trying to reproduce this problem based on descriptions in emails. For the sake of the sanity of our committers, these kind of problems should be accompanied by a pull request that contains a script that

Re: How do I send binary data to avatica?

Ok, I will try to circle around to this. Meanwhile, let's start thinking about ways that you and I can better collaborate :) We want to make building drivers for Avatica in other languages as painless as possible, but it seems like we have a big impedance mismatch already -- not having a

Re: How do I send binary data to avatica?

Forgot to add: Yes, still using Hbase 1.1.4 with phoenix 4.7.0. On 18/04/2016 3:31 AM, Josh Elser wrote: Sorry, yes. I just didn't want to have to post the JSON elsewhere -- it would've been gross to include in an email. For JSON, it is base64 encoded. This is not necessary for protobuf (which

Re: How do I send binary data to avatica?

I have now created a repo on github containing all the binary protobuf requests and responses to insert binary data into a VARBINARY column. The repo is available here: https://github.com/F21/avatica-binary-protobufs The file 8-request is of most interest because that's the one where we are

Re: How do I send binary data to avatica?

I just tried using sqlline-thin and it's also showing that the binary values are not inserted. Interestingly, from my experiments, if I send a STRING to update a binary column, it works. If I send a BYTE_STRING, it doesn't work. Would it help if I create a repo on github with containing the

Re: How do I send binary data to avatica?

Sorry, yes. I just didn't want to have to post the JSON elsewhere -- it would've been gross to include in an email. For JSON, it is base64 encoded. This is not necessary for protobuf (which can natively handle raw bytes). For #2, did you verify that the data made it into the database

Re: How do I send binary data to avatica?

Just reporting back my experiments: 1. Using JSON: I set the serialization of the query server to JSON and replayed your requests using CURL. For the binary file, I first base64 encode it into a string and then sent it. This worked properly and I can see data inserted into the table using

Re: How do I send binary data to avatica?

Hey Josh, I am just spinning up my docker containers to test using JSON first. You have shown the binary data as but do I need to base64 encode them for JSON? On 17/04/2016 8:17 AM, Josh Elser wrote: I wrote a simple test case for this with the gopher image. Here's the JSON data (but

Re: How do I send binary data to avatica?

I wrote a simple test case for this with the gopher image. Here's the JSON data (but hopefully this is enough to help out). I'd have to write more code to dump out the actual protobufs. Let me know if this is insufficient to help you figure out what's wrong. My test worked fine. CREATE TABLE

Re: How do I send binary data to avatica?

Hey Josh, Here's what I am doing: Create the table: CREATE TABLE test ( int INTEGER PRIMARY KEY, bin VARBINARY) TRANSACTIONAL=false Download the binary file we want to insert from here: https://raw.githubusercontent.com/golang-samples/gopher-vector/master/gopher.png Prepare the statement:

Re: How do I send binary data to avatica?

Yeah, that sounds right to me too. I think we have a test for random bytes. Maybe there's something weird happening under the hood in the Avatica JDBC driver that isn't obvious to you in the Go driver. Any chance you can share some example code you're running? I can try to convert it to a

Re: How do I send binary data to avatica?

I also tried casting the data to a string and setting it to StringValue and the Rep type to STRING. This works when I store and retrieve strings from the binary column, but doesn't work correctly if I try to store something like a small image. On 14/04/2016 5:03 PM, Julian Hyde wrote:

Re: How do I send binary data to avatica?

BytesValue sounds right. I’m not sure why it isn’t working for you. > On Apr 14, 2016, at 6:34 AM, F21 wrote: > > As mentioned on the list, I am currently working on a golang client/driver > for avatica using protobufs for serialization. > > I've got all datatypes