[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477296#comment-16477296
]
Andras Piros commented on OOZIE-3250:
-
Thanks [~gezapeti]! Failing tests are unrelated.
Committed to {{master}}.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch,
> OOZIE-3250.006.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477276#comment-16477276
]
Hadoop QA commented on OOZIE-3250:
--
Testing JIRA OOZIE-3250
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2132
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)
.Tests failing with errors:
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/540/
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch,
> OOZIE-3250.006.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477210#comment-16477210
]
Peter Cseh commented on OOZIE-3250:
---
+1 (pending Jenkins on patch-6). The latest jhenkins run I could find is for
patch-5.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch,
> OOZIE-3250.006.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477198#comment-16477198
]
Hadoop QA commented on OOZIE-3250:
--
PreCommit-OOZIE-Build started
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch,
> OOZIE-3250.006.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477192#comment-16477192
]
Andras Piros commented on OOZIE-3250:
-
Thanks for the reviews [~gezapeti] [~mi...@cloudera.com]! Renamed to
{{ByteArrayUtils}}, and extended Javadoc by mentioning the performance numbers
from patch 004.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch,
> OOZIE-3250.006.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476293#comment-16476293
]
Misha Dmitriev commented on OOZIE-3250:
---
The last patch looks good to me. Thank you for your efforts, [~andras.piros]
It's great that you tested performance of this stuff. I realize that the
performance measurement code is fragile so it doesn't make sense to keep it as
a test that gets re-run regularly. But I think it would make sense to at least
mention your performance numbers in the javadoc of the ByteArrayUtils class, so
that in the future people don't have doubts about it.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475946#comment-16475946
]
Hadoop QA commented on OOZIE-3250:
--
Testing JIRA OOZIE-3250
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2130
.Tests failed: 17
.Tests errors: 12
.The patch failed the following testcases:
testPartitionDependency(org.apache.oozie.service.TestPartitionDependencyManagerEhcache)
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
testCoordinatorActionSelectors(org.apache.oozie.jms.TestJMSJobEventListener)
testCoordinatorActionSelectorsNegative(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectors(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorJobSuccessEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobSuspendEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectorsAnd(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorJobFailureEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectorsNegative(org.apache.oozie.jms.TestJMSJobEventListener)
testWorkflowJobSelectorsOr(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobSuccessEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorActionWaitingEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobFailureEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnCoordinatorActionStartEvent(org.apache.oozie.jms.TestJMSJobEventListener)
testOnWorkflowJobStartedEvent(org.apache.oozie.jms.TestJMSJobEventListener)
.Tests failing with errors:
testDropEventTypeMessage(org.apache.oozie.jms.TestHCatMessageHandler)
testCacheUpdateByMessage(org.apache.oozie.jms.TestHCatMessageHandler)
testOnSLADurationMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAStartMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testSLAJobSelectorsNegative(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAEndMetEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAStartMissEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testSLAJobSelectors(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLAEndMissEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testOnSLADurationMissEvent(org.apache.oozie.jms.TestJMSSLAEventListener)
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
testThreadLocalSession(org.apache.oozie.jms.TestDefaultConnectionContext)
.
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475840#comment-16475840
]
Hadoop QA commented on OOZIE-3250:
--
Testing JIRA OOZIE-3250
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2131
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)
.Tests failing with errors:
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/533/
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475819#comment-16475819
]
Hadoop QA commented on OOZIE-3250:
--
PreCommit-OOZIE-Build started
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475807#comment-16475807
]
Andras Piros commented on OOZIE-3250:
-
Removed also {{TestByteUtils#testInterningRandomByteArrays()}}.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch, OOZIE-3250.005.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475709#comment-16475709
]
Hadoop QA commented on OOZIE-3250:
--
PreCommit-OOZIE-Build started
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475702#comment-16475702
]
Andras Piros commented on OOZIE-3250:
-
[~gezapeti] removed
{{TestByteUtils#testInterningSameByteArrayManyTimesDoesNotIncreaseHeap()}} and
introduced {{TestByteUtils#testInterningRandomByteArrays()}} that does 10k *
10k random {{byte[]}} allocation and interning. Allocation alone takes around
7.8 seconds on my box, allocation plus interning takes around 8.0 seconds -
interning doesn't seem to take too many resources both in terms of CPU and
memory.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch, OOZIE-3250.004.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475656#comment-16475656
]
Hadoop QA commented on OOZIE-3250:
--
Testing JIRA OOZIE-3250
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2131
.Tests failed: 1
.Tests errors: 2
.The patch failed the following testcases:
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
.Tests failing with errors:
testConnectionRetryExceptionListener(org.apache.oozie.service.TestJMSAccessorService)
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/532/
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475627#comment-16475627
]
Peter Cseh commented on OOZIE-3250:
---
Thanks for the fixes [~andras.piros]!
I get itchy when I look at
testInterningSameByteArrayManyTimesDoesNotIncreaseHeap(). It feels like this
will be the next flaky test we're chasing. I don't think it's worth keeping. I
still have a dream when we're capable of running tests in parallel and this
test is probably one working against that. Please remove it.
If weakIntern() works as I think it does it may get slow after it have
internalized a lot of arrays. Can you do a measurement of 10k byte[]s of random
lengths but the same content (as an easily coded worst-case-like scenario)? I
don't think it will count but still, things will get slower if the number of
internalized arrays is big.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475541#comment-16475541
]
Hadoop QA commented on OOZIE-3250:
--
PreCommit-OOZIE-Build started
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch,
> OOZIE-3250.003.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475528#comment-16475528
]
Andras Piros commented on OOZIE-3250:
-
[~gezapeti] using now {{==}} instead of {{Arrays.equals()}} to show that all
interned {{byte[]}} instances are the same. Also introduced another test case
to show that interning a huge {{byte[]}} many times doesn't increase heap in
any significant way, and is fast (1000 x 1MB in 1.5 seconds on my machine). So
I don't think we have a performance problem here. I'm also sure the contents of
the interned {{byte[]}} instances aren't changing without modifying the
underlying arrays. I'm planning another round of Oozie heap analysis with
multiple running coordinators before and after the fix.
[~mi...@cloudera.com] added a similar check before to show that copied
{{byte[]}} point to another instances, for better maintainability.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474705#comment-16474705
]
Misha Dmitriev commented on OOZIE-3250:
---
I agree with [~gezapeti] wrt. the needed changes to the test. Arrays.equals()
compares the _contents_ of two arrays, i.e. we can have ar1 != ar2, but
Arrays.equals(ar1, ar2). So it's exactly opposite of what we need in this test,
because we really want to achieve ar1 == ar2 after internalization of ar1 and
ar2. So please replace Arrays.equals() with '==' and maybe furthermore add a
similar check, but with !=, for arrays before internalization, same as done in
my original example. This will hopefully remove some confusion for the future
maintainers of this code.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473992#comment-16473992
]
Peter Cseh commented on OOZIE-3250:
---
Thanks for the improvement [~andras.piros]!
Shouldn't we use == in the test instead ofArrays.equals ? This test passes
even if I remove internalization from the code.
Any ideas on the performance impact of reading through all the byte arrays? I
don't imagine it's huge as we'll usually parse there arrays as Strings later.
Would it make sense to keep only one copy of the data in StringBlob and
BinaryBlob? E.g. when the getString is called in StringBlob, the byte array
could be nulled out as it can be recalculated if needed from the String value.
This would again be some calculation overhead added to gain some space in
memory. I'm not sure it's worth it but it might worth a try.
Also, are we 100% sure that the contents of the arrays are not changing? I
haven't checked all the places in all the Beans but it looks like we're in the
safe zone here.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473971#comment-16473971
]
Hadoop QA commented on OOZIE-3250:
--
Testing JIRA OOZIE-3250
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2130
.Tests failed: 1
.Tests errors: 1
.The patch failed the following testcases:
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
.Tests failing with errors:
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/529/
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473895#comment-16473895
]
Andras Piros commented on OOZIE-3250:
-
Thanks [~mi...@cloudera.com] for your insights!
Please find attached the new patch, addressing this {{equals()}} / {{==}} issue
w/ {{byte[]}} instances. In this implementation I used {{java.nio.ByteBuffer}}.
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473896#comment-16473896
]
Hadoop QA commented on OOZIE-3250:
--
PreCommit-OOZIE-Build started
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch, OOZIE-3250.002.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471018#comment-16471018
]
Misha Dmitriev commented on OOZIE-3250:
---
Thank you for making this change, [~andras.piros]!
I was on the verge of approving it when I realized that attempting to intern
arrays in the same way as non-array objects may not work as expected. That's
because the equals() method in arrays compares two arrays with a simple '==',
rather than by contents. So you have to use some more code to achieve what you
want. See the test below that I wrote, it's self-explanatory enough.
{code:java}
import com.google.common.collect.Interner;
import com.google.common.collect.Interners;
import org.junit.Assert;
import org.junit.Test;
import java.util.Arrays;
public class TestByteArrayInterning {
private static final Interner BYTE_ARRAY_INTERNER =
Interners.newWeakInterner();
private static final Interner BYTE_ARWR_INTERNER =
Interners.newWeakInterner();
@Test
public void testByteArrayInterning() {
int nArrs = 1000;
int nEls = 100;
byte[][] sameContentArrs = new byte[nArrs][];
for (int i = 0; i < nArrs; i++) {
byte[] b = new byte[nEls];
sameContentArrs[i] = b;
for (int j = 0; j < nEls; j++) b[j] = (byte) j;
}
byte[][] internedArrs = new byte[nArrs][];
for (int i = 0; i < nArrs; i++) {
internedArrs[i] = BYTE_ARRAY_INTERNER.intern(sameContentArrs[i]);
}
// Trying to intern byte[] arrays directly doesn't work, because in the
implemenation
// of arrays the equals() method is the same as '=='. It doesn't compare
arrays' _contents_
for (int i = 1; i < nArrs; i++) {
Assert.assertFalse(internedArrs[i-1] == internedArrs[i]);
}
for (int i = 0; i < nArrs; i++) {
internedArrs[i] = BYTE_ARWR_INTERNER.intern(new
ByteArrayWrapper(sameContentArrs[i])).bytes;
}
for (int i = 1; i < nArrs; i++) {
Assert.assertTrue(internedArrs[i-1] == internedArrs[i]);
}
}
/**
* A wrapper class, whose only purpose is to effectively override the wrapped
* array's equals() method, so that we compare the contents of two arrays.
* The default implementation of equals() in an array performs just an
identity
* check.
*
* If arrays are big and/or CPU performance of this code is critical, one may
* consider caching the hashcode of the wrapped array after hashCode() is
called
* for the first time.
*/
private static class ByteArrayWrapper {
private final byte[] bytes;
ByteArrayWrapper(byte[] bytes) {
this.bytes = bytes;
}
@Override
public int hashCode() {
return Arrays.hashCode(bytes);
}
@Override
public boolean equals(Object other) {
if (!(other instanceof ByteArrayWrapper)) return false;
return Arrays.equals(bytes, ((ByteArrayWrapper) other).bytes);
}
}
}{code}
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470306#comment-16470306
]
Andras Piros commented on OOZIE-3250:
-
Failing tests are unrelated. [~mi...@cloudera.com] [~gezapeti] can you please
review?
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470292#comment-16470292
]
Hadoop QA commented on OOZIE-3250:
--
Testing JIRA OOZIE-3250
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2129
.Tests failed: 1
.Tests errors: 2
.The patch failed the following testcases:
testCoordActionRecoveryServiceForWaitingRegisterPartition(org.apache.oozie.service.TestRecoveryService)
.Tests failing with errors:
testConnectionRetryExceptionListener(org.apache.oozie.service.TestJMSAccessorService)
testConnectionRetry(org.apache.oozie.service.TestJMSAccessorService)
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/524/
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3250) Reduce heap waste by reducing duplicate byte[] count
[
https://issues.apache.org/jira/browse/OOZIE-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470194#comment-16470194
]
Hadoop QA commented on OOZIE-3250:
--
PreCommit-OOZIE-Build started
> Reduce heap waste by reducing duplicate byte[] count
>
>
> Key: OOZIE-3250
> URL: https://issues.apache.org/jira/browse/OOZIE-3250
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3250.001.patch
>
>
> Similar to OOZIE-3232, we also need to intern the {{byte[]}} field values
> within
> [*{{BinaryBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BinaryBlob.java#L32-L33]
> and
> [*{{StringBlob}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/StringBlob.java#L34]
> to reduce heap waste caused by duplicate {{byte[]}} entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
