Thanks Josh for the response!
I would definitely like to use protobuf serialization, but I'm observing
performance issues trying to run queries with a large number of results.
One problem is that I observe PQS runs out of memory, when its trying to
(what looks like to me) serialize the results in Avatica. The other is
that the phoenixdb python adapter itself spends a large amount of time
in the logic
<https://github.com/apache/phoenix/blob/master/python/phoenixdb/phoenixdb/cursor.py#L248>
where its converting the protobuf rows to python objects.
Interestingly when we use sqlline-thin.py instead of python phoenixdb,
the protobuf serialization works fine and responses are fast. It's not
clear to me why PQS would have problems when using the python adapter
and not when using sqlline-thin, do they follow different code paths
(especially around serialization)?
Thanks again,
Manoj
On Fri, Nov 2, 2018 at 4:05 PM Josh Elser <els...@apache.org
<mailto:els...@apache.org>> wrote:
I would strongly suggest you do not use the JSON serialization.
The JSON support is implemented via Jackson which has no means to make
backwards compatibility "easy". On the contrast, protobuf makes this
extremely easy and we have multiple examples over the past years where
we've been able to fix bugs in a backwards compatible manner.
If you want the thin client to continue to work across versions, stick
with protobuf.
On 11/2/18 5:27 PM, Manoj Ganesan wrote:
> Hey everyone,
>
> I'm trying to use the Python phoenixdb adapter work with JSON
> serialization on PQS.
>
> I'm using Phoenix 4.14 and the adapter works fine with protobuf, but
> when I try making it work with an older version of phoenixdb
(before the
> JSON to protobuf switch was introduced), it just returns 0 rows.
I don't
> see anything in particular wrong with the HTTP requests itself,
and they
> seem to conform to the Avatica JSON spec
> (http://calcite.apache.org/avatica/docs/json_reference.html).
>
> Here's the result (with some debug statements) that returns 0 rows.
> Notice the *"firstFrame":{"offset":0,"done":true,"rows":[]* below:
>
> request body = {"maxRowCount": -2, "connectionId":
> "68c05d12-5770-47d6-b3e4-dba556db4790", "request":
"prepareAndExecute",
> "statementId": 3, "sql": "SELECT col1, col2 from table limit 20"}
> request headers = {'content-type': 'application/json'}
> _post_request: got response {'fp': <socket._fileobject object at
> 0x7f858330b9d0>, 'status': 200, 'will_close': False, 'chunk_left':
> 'UNKNOWN', 'length': 1395, 'strict': 0, 'reason': 'OK',
'version': 11,
> 'debuglevel': 0, 'msg': <httplib.HTTPMessage instance at
> 0x7f84fb50be18>, 'chunked': 0, '_method': 'POST'}
> response.read(): body =
>
{"response":"executeResults","missingStatement":false,"rpcMetadata":{"response":"rpcMetadata","serverAddress":"ip-10-55-6-247:8765"},"results":[{"response":"resultSet","connectionId":"68c05d12-5770-47d6-b3e4-dba556db4790","statementId":3,"ownStatement":true,"signature":{"columns":[{"ordinal":0,"autoIncrement":false,"caseSensitive":false,"searchable":true,"currency":false,"nullable
>
":0,"signed":true,"displaySize":40,"label":"COL1","columnName":"COL1","schemaName":"","precision":0,"scale":0,"tableName":"TABLE","catalogName":"","type":{"type":"scalar","id":4,"name":"INTEGER","rep":"PRIMITIVE_INT"},"readOnly":true,"writable":false,"definitelyWritable":false,"columnClassName":"java.lang.Integer"},{"ordinal":1,"autoIncrement":false,"caseSensitive":false,"searchable":true,"currency":false,"nullable":0,"signed":true,"displaySize":40,"label":"COL2","columnName":"COL2","schemaName":"","precision":0,"scale":0,"tableName":"TABLE","catalogName":"","type":{"type":"scalar","id":4,"name":"INTEGER","rep":"PRIMITIVE_INT"},"readOnly":true,"writable":false,"definitelyWritable":false,"columnClassName":"java.lang.Integer"}],"sql":null,"parameters":[],"cursorFactory":{"style":"LIST","clazz":null,"fieldNames":null},"statementType":null},*"firstFrame":{"offset":0,"done":true,"rows":[]*},"updateCount":-1,"rpcMetadata":{"response":"rpcMetadata","serverAddress":"ip-10-55-6-247:8765"}}]}
>
>
> The same query issued against a PQS started with PROTOBUF
serialization
> and using a newer phoenixdb adapter returns the correct number of
rows.
>
> Has anyone had luck making this work?
>
> Thanks,
> Manoj
>
