[ https://issues.apache.org/jira/browse/IMPALA-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758590#comment-16758590 ]
ASF subversion and git services commented on IMPALA-7934: --------------------------------------------------------- Commit b0942296ab5f24660473abc218d45978fc402d81 in impala's branch refs/heads/master from Fredy Wijaya [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b094229 ] IMPALA-7934: Switch to java.util.Base64 implementation It is shown that java.util.Base64 implementation seems to have better performance compared to Apache Commons Codec's Base64 implementation, which can benefit operations, such as incremental stats. This patch switches the implementation of Base64 from Apache Commons Codec to java.util.Base64 implementation. This is the JMH benchmark result comparing java.util.Base64 vs Commons Codec's Base64: Result "base64.Base64Benchmark.javaBase64": 31.149 ±(99.9%) 1.567 ms/op [Average] (min, avg, max) = (27.564, 31.149, 34.675), stdev = 2.091 CI (99.9%): [29.583, 32.716] (assumes normal distribution) Result "base64.Base64Benchmark.codecBase64": 65.921 ±(99.9%) 4.762 ms/op [Average] (min, avg, max) = (58.072, 65.921, 80.470), stdev = 6.357 CI (99.9%): [61.159, 70.683] (assumes normal distribution) Benchmark Mode Cnt Score Error Units Base64Benchmark.javaBase64 avgt 25 31.149 ± 1.567 ms/op Base64Benchmark.codecBase64 avgt 25 65.921 ± 4.762 ms/op Testing: - Ran all FE tests - Created a table with incremental stats without a patch and read it with the patch Change-Id: I2d43d4a4f073a800d963ce4c77f21c9efa8471ac Reviewed-on: http://gerrit.cloudera.org:8080/12250 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Switch to using Java 8's Base64 impl for incremental stats encoding > ------------------------------------------------------------------- > > Key: IMPALA-7934 > URL: https://issues.apache.org/jira/browse/IMPALA-7934 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Affects Versions: Impala 3.1.0 > Reporter: bharath v > Assignee: Fredy Wijaya > Priority: Major > Labels: ramp-up > Fix For: Impala 3.2.0 > > Attachments: base64.png > > > Incremental stats are compressed and Base64 encoded before they are chunked > and written to the HMS' partition parameters map. When they are read back, we > need to Base64 decode and decompress. > For certain incremental stats heavy tables, we noticed that a significant > amount of time is spent in these base64 classes (see the attached image for > the stack. Unfortunately, I don't have the text version of it). > Java 8 comes with its own Base64 implementation and that has shown much > better perf results [1] compared to apache codec's impl. So consider > switching to Java 8's base64 impl. > [1] http://java-performance.info/base64-encoding-and-decoding-performance/ > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org