[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124953#comment-17124953
]
Attila Magyar edited comment on HIVE-23277 at 6/3/20, 1:31 PM:
---------------------------------------------------------------
Hey [~rajesh.balamohan], I made a patch for this, where json serialization
happens on the logWriter's thread. The event is only built partially up front
with a json object (not the serialized string) and the conversion happens right
before writing out the event.
However events like this take up more space in memory as before. About twice as
much. The queue has a default max capacity of 64 so this might not be a problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX with JSON object: 779536{code}
How significant do you think the speed improvements is? Is it worth it? Based
on my own measurements the JSON serialization wasn't that slow with the queries
I used (about 10-15 ms).
was (Author: amagyar):
Hey [~rajesh.balamohan], I made a patch for this, where json serialization
happens on the logWriter's thread. The event is only built partially up front
with a json object (not the serialized string) and the conversion happens right
before writing out the event.
However events like this takes up more space in memory as before. About twice
as much. The queue has a default max capacity of 64 so this might not be a
problem.
{code:java}
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.838Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX size with serialized JSON: 392288
./app=hiveserver2/2020-06-03-13-05_0.log.gz:<14>1 2020-06-03T13:07:39.833Z
hiveserver2-0.hiveserver2-service.compute-1591188147-6npj.svc.cluster.local
hiveserver2 1 7e79dde9-4ac7-4df6-932f-1be75ec58e73 [mdc@18060
class="hooks.HiveProtoLoggingHook" level="INFO" thread="Hive Hook Proto Log
Writer 0"] XXX with JSON object: 779536{code}
> HiveProtoLogger should carry out JSON conversion in its own thread
> ------------------------------------------------------------------
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Attila Magyar
> Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
