[
https://issues.apache.org/jira/browse/HIVE-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744918#comment-17744918
]
László Bodor commented on HIVE-27511:
-------------------------------------
hey [~zabetak], thanks for the input! basically, all of your suggestions make
sense, even Hadoop level, however, I feel we might want to make this for hive /
tez purpose in the first round, and I think we should target both:
1. *hive* side (current patch): print insights about the payload passed to
different vertices in serialization time in a costly manner: if the insights
are turned on, we need to sort the config object and print top 10, which I
don't want to do for every query: this is hive specific and can be turned on
and off, so it can become even more costly over time
2. *tez* size: a more general one, simply log the config string length if it's
beyond a certain length: this is quite cheap, calling a String.length() and log
a message if it's beyond a certain amount (configurable)
what do you think?
> Print Tez processor payload insights
> ------------------------------------
>
> Key: HIVE-27511
> URL: https://issues.apache.org/jira/browse/HIVE-27511
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
>
> When investigating TezAM OOM issues, a common problem is a huge conf. In this
> we usually do the following:
> 1. ask for heapdump
> 2. analyze heapdump, and finally find huge UserPayload, which is a binary :)
> extremely hard to read
> 3. deserialize userpayload
> 4. looks for configuration properties in payload to blame
> 5. figure out how to reduce the payload
> I think with a handy configuration insight log message in HS2, we need to do
> only 5) (as 1) 2) 3) 4) can be complicated)
> with the initial implementation I got this with e.g. ptf.q
> {code}
> 2023-07-18T05:02:16,080 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: Payload size for Map 1: 82967 bytes, # of
> keys in conf: 2653
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 1.
> '/Users/laszlobodor/apache/hive/itests/qtest/target/tmp/localscratchdir/8e476599-63fd-4530-a007-71d92afced89/hive_2023-07-18_05-02-15_122_2130403664877939930-1/laszlobodor/_tez_scratch_dir/2d7f67a1-b7a3-413d-a027-51978f85e862/map.xml'
> size: 2552
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 2. 'hive.conf.hidden.list' size: 526
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 3. 'hadoop.security.sensitive-config-keys'
> size: 496
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 4. 'hive.serdes.using.metastore.for.schema'
> size: 426
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 5. 'fs.s3a.aws.credentials.provider' size: 252
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 6. 'hive.exec.plan' size: 229
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 7. 'io.serializations' size: 180
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 8. 'hive.exec.post.hooks' size: 177
> 2023-07-18T05:02:16,084 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 9. '_hive_tez_tmp_dir' size: 173
> 2023-07-18T05:02:16,085 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 10.
> 'dfs.webhdfs.acl.provider.permission.pattern' size: 154
> ...
> 2023-07-18T05:02:19,051 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: Payload size for Reducer 2: 90231 bytes, # of
> keys in conf: 2658
> 2023-07-18T05:02:19,052 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 1.
> '/Users/laszlobodor/apache/hive/itests/qtest/target/tmp/localscratchdir/8e476599-63fd-4530-a007-71d92afced89/hive_2023-07-18_05-02-18_172_1139677043401490299-1/laszlobodor/_tez_scratch_dir/040836dc-01a2-438b-99ff-148d1f5afd48/reduce.xml'
> size: 9364
> 2023-07-18T05:02:19,052 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 2. 'hive.conf.hidden.list' size: 526
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 3. 'hadoop.security.sensitive-config-keys'
> size: 496
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 4. 'hive.serdes.using.metastore.for.schema'
> size: 426
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 5. 'columns.types' size: 377
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 6. 'columns' size: 367
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 7. 'fs.s3a.aws.credentials.provider' size: 252
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 8. 'hive.exec.plan' size: 229
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 9. 'io.serializations' size: 180
> 2023-07-18T05:02:19,053 INFO [8e476599-63fd-4530-a007-71d92afced89 Listener
> at 0.0.0.0/55258] tez.DagUtils: 10. 'hive.exec.post.hooks' size: 177
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
