[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[ https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085066#comment-16085066 ] Thomas D'Silva commented on PHOENIX-3978: - I made a change to see if the test works that I forgot to revert on the 4.x-HBase-0.98 branch, I have committed a fix. > Expose mutation failures in our metrics > --- > > Key: PHOENIX-3978 > URL: https://issues.apache.org/jira/browse/PHOENIX-3978 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Thomas D'Silva > Fix For: 4.12.0 > > Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch, > PHOENIX-3978-4.x-HBase-0.98-v3.patch, PHOENIX-3978.patch, > PHOENIX-3978-v2.patch > > > We should be exposing whether a mutation has failed through our metrics > system. This should be done both within global and request level metrics. > The task basically boils down to: > 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType. > 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in > GlobalClientMetrics > 3) Adding a new CombinableMetric member called mutationBatchFailed to > MutationMetric class > 4) Making sure that the two metrics are updated within the catch exception > block of MutationState#send() > 5) Unit test in PhoenixMetricsIT > FYI, [~tdsilva] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[
https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085048#comment-16085048
]
Samarth Jain commented on PHOENIX-3978:
---
[~tdsilva], looks like your checkin broke unit tests.
{code}
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.019 s
<<< FAILURE! - in org.apache.phoenix.metrics.MetricTypeTest
[ERROR] testUniqueShortNames(org.apache.phoenix.metrics.MetricTypeTest) Time
elapsed: 0.018 s <<< FAILURE!
java.lang.AssertionError: Metric short names should be unique found duplicates
for MEMORY_CHUNK_BYTES and MUTATION_BATCH_SIZE
at
org.apache.phoenix.metrics.MetricTypeTest.testUniqueShortNames(MetricTypeTest.java:37)
{code}
> Expose mutation failures in our metrics
> ---
>
> Key: PHOENIX-3978
> URL: https://issues.apache.org/jira/browse/PHOENIX-3978
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Thomas D'Silva
> Fix For: 4.12.0
>
> Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch,
> PHOENIX-3978-4.x-HBase-0.98-v3.patch, PHOENIX-3978.patch,
> PHOENIX-3978-v2.patch
>
>
> We should be exposing whether a mutation has failed through our metrics
> system. This should be done both within global and request level metrics.
> The task basically boils down to:
> 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType.
> 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in
> GlobalClientMetrics
> 3) Adding a new CombinableMetric member called mutationBatchFailed to
> MutationMetric class
> 4) Making sure that the two metrics are updated within the catch exception
> block of MutationState#send()
> 5) Unit test in PhoenixMetricsIT
> FYI, [~tdsilva]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[ https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084841#comment-16084841 ] Hudson commented on PHOENIX-3978: - FAILURE: Integrated in Jenkins build Phoenix-master #1683 (See [https://builds.apache.org/job/Phoenix-master/1683/]) PHOENIX-3978 Expose mutation failures in our metrics (thomas: rev e05b7cdd3286d7021d5836ecf7446a1c2aed0b9c) * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/CombinableMetric.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixResultSet.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixMetricsIT.java * (add) phoenix-core/src/test/java/org/apache/phoenix/metrics/MetricTypeTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MutationMetricQueue.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/OverAllQueryMetrics.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/NonAtomicMetric.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/execute/PartialCommitIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricType.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ReadMetricQueue.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/AtomicMetric.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalClientMetrics.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/Metric.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/PhoenixRuntime.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/CombinableMetricImpl.java > Expose mutation failures in our metrics > --- > > Key: PHOENIX-3978 > URL: https://issues.apache.org/jira/browse/PHOENIX-3978 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Thomas D'Silva > Fix For: 4.12.0 > > Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch, > PHOENIX-3978-4.x-HBase-0.98-v3.patch, PHOENIX-3978.patch, > PHOENIX-3978-v2.patch > > > We should be exposing whether a mutation has failed through our metrics > system. This should be done both within global and request level metrics. > The task basically boils down to: > 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType. > 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in > GlobalClientMetrics > 3) Adding a new CombinableMetric member called mutationBatchFailed to > MutationMetric class > 4) Making sure that the two metrics are updated within the catch exception > block of MutationState#send() > 5) Unit test in PhoenixMetricsIT > FYI, [~tdsilva] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[
https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084463#comment-16084463
]
Hadoop QA commented on PHOENIX-3978:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12876913/PHOENIX-3978-4.x-HBase-0.98-v3.patch
against 4.x-HBase-0.98 branch at commit
caeaec8355023ce93656f56ad6dfa77d5dbc3517.
ATTACHMENT ID: 12876913
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1202//console
This message is automatically generated.
> Expose mutation failures in our metrics
> ---
>
> Key: PHOENIX-3978
> URL: https://issues.apache.org/jira/browse/PHOENIX-3978
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch,
> PHOENIX-3978-4.x-HBase-0.98-v3.patch, PHOENIX-3978.patch,
> PHOENIX-3978-v2.patch
>
>
> We should be exposing whether a mutation has failed through our metrics
> system. This should be done both within global and request level metrics.
> The task basically boils down to:
> 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType.
> 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in
> GlobalClientMetrics
> 3) Adding a new CombinableMetric member called mutationBatchFailed to
> MutationMetric class
> 4) Making sure that the two metrics are updated within the catch exception
> block of MutationState#send()
> 5) Unit test in PhoenixMetricsIT
> FYI, [~tdsilva]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[ https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084470#comment-16084470 ] Samarth Jain commented on PHOENIX-3978: --- +1 > Expose mutation failures in our metrics > --- > > Key: PHOENIX-3978 > URL: https://issues.apache.org/jira/browse/PHOENIX-3978 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Thomas D'Silva > Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch, > PHOENIX-3978-4.x-HBase-0.98-v3.patch, PHOENIX-3978.patch, > PHOENIX-3978-v2.patch > > > We should be exposing whether a mutation has failed through our metrics > system. This should be done both within global and request level metrics. > The task basically boils down to: > 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType. > 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in > GlobalClientMetrics > 3) Adding a new CombinableMetric member called mutationBatchFailed to > MutationMetric class > 4) Making sure that the two metrics are updated within the catch exception > block of MutationState#send() > 5) Unit test in PhoenixMetricsIT > FYI, [~tdsilva] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[
https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084267#comment-16084267
]
Samarth Jain commented on PHOENIX-3978:
---
Thanks for the updated patch, [~tdsilva]. Looks good. One comment:
I am not sure if this test is doing the right thing.
{code}
@Test
+public void testUniqueShortNames() throws Exception {
+ Map shortNameMap =
Maps.newHashMapWithExpectedSize(MetricType.values().length);
+for (MetricType type : MetricType.values()) {
+ if (shortNameMap.containsKey(type.shortName())) {
+ fail("Metric short names should be unique found
duplicates for " + type.name() + " and "
+ +
shortNameMap.get(type.shortName()).name());
+ }
+}
+}
{code}
The if part should read like this?
{code}
if (shortNameMap.put(type.shortName(), type) != null) {
{code}
> Expose mutation failures in our metrics
> ---
>
> Key: PHOENIX-3978
> URL: https://issues.apache.org/jira/browse/PHOENIX-3978
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch,
> PHOENIX-3978.patch, PHOENIX-3978-v2.patch
>
>
> We should be exposing whether a mutation has failed through our metrics
> system. This should be done both within global and request level metrics.
> The task basically boils down to:
> 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType.
> 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in
> GlobalClientMetrics
> 3) Adding a new CombinableMetric member called mutationBatchFailed to
> MutationMetric class
> 4) Making sure that the two metrics are updated within the catch exception
> block of MutationState#send()
> 5) Unit test in PhoenixMetricsIT
> FYI, [~tdsilva]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[
https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083373#comment-16083373
]
Hadoop QA commented on PHOENIX-3978:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12876720/PHOENIX-3978-v2.patch
against master branch at commit b7b571b7db0c58ff488e435d5a3cf6c45a41fe86.
ATTACHMENT ID: 12876720
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
50 warning messages.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+Map> mutationWriteMetrics =
PhoenixRuntime.getWriteMetricInfoForMutationsSinceLastReset(con);
+assertEquals(expectedUncommittedStatementIndexes.length,
mutationWriteMetrics.get(B_FAILURE_TABLE).get(MUTATION_BATCH_FAILED_SIZE).intValue());
+assertEquals(expectedUncommittedStatementIndexes.length,
GLOBAL_MUTATION_BATCH_FAILED_COUNT.getMetric().getTotalSum());
+Map> mutationMetrics =
PhoenixRuntime.getWriteMetricInfoForMutationsSinceLastReset(pConn);
+Map> readMetrics =
PhoenixRuntime.getReadMetricInfoForMutationsSinceLastReset(pConn);
+Map> mutationMetrics =
PhoenixRuntime.getWriteMetricInfoForMutationsSinceLastReset(pConn);
+Map> readMetrics =
PhoenixRuntime.getReadMetricInfoForMutationsSinceLastReset(pConn);
+Map> mutationMetrics =
PhoenixRuntime.getWriteMetricInfoForMutationsSinceLastReset(pConn);
+Map> readMetrics =
PhoenixRuntime.getReadMetricInfoForMutationsSinceLastReset(pConn);
+Map> readMetrics =
PhoenixRuntime.getRequestReadMetricInfo(rs);
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ImmutableIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ImmutableIndexWithStatsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.UpsertSelectIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SaltedViewIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ViewIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.hadoop.hbase.regionserver.wal.WALReplayWithIndexWritesAndCompressedWALIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.NotQueryIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ClientTimeArithmeticQueryIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1197//testReport/
Javadoc warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1197//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1197//console
This message is automatically generated.
> Expose mutation failures in our metrics
> ---
>
> Key: PHOENIX-3978
> URL: https://issues.apache.org/jira/browse/PHOENIX-3978
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3978-4.x-HBase-0.98-v2.patch,
> PHOENIX-3978.patch, PHOENIX-3978-v2.patch
>
>
> We should be exposing whether a mutation has failed through our metrics
> system. This should be done both within global and request level metrics.
> The task basically boils down to:
> 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType.
> 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in
> GlobalClientMetrics
> 3) Adding a new CombinableMetric member called mutationBatchFailed to
> MutationMetric class
> 4) Making sure that the two metrics are updated within the catch exception
> block of MutationState#send()
> 5) Unit test in PhoenixMetricsIT
> FYI, [~tdsilva]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-3978) Expose mutation failures in our metrics
[
https://issues.apache.org/jira/browse/PHOENIX-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082793#comment-16082793
]
Samarth Jain commented on PHOENIX-3978:
---
Thanks for the patch, [~tdsilva]. Looks good for the most part. A couple of
comments:
In PartialCommitIT, maybe add a line to test global mutation failed metric too?
{code}
+Map> mutationWriteMetrics =
PhoenixRuntime.getWriteMetricsForMutationsSinceLastReset(con);
+assertEquals(expectedUncommittedStatementIndexes.length,
mutationWriteMetrics.get(B_FAILURE_TABLE).get(MUTATION_BATCH_FAILED_COUNT).intValue());
{code}
For backward compatibility, we should leave the old PhoenixRuntime and
corresponding aggregate() methods in the metric queues. We should mark them as
deprecated with a note to remove them in the next major release.
> Expose mutation failures in our metrics
> ---
>
> Key: PHOENIX-3978
> URL: https://issues.apache.org/jira/browse/PHOENIX-3978
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Thomas D'Silva
> Attachments: PHOENIX-3978.patch
>
>
> We should be exposing whether a mutation has failed through our metrics
> system. This should be done both within global and request level metrics.
> The task basically boils down to:
> 1) Adding a new enum MUTATION_BATCH_FAILED_COUNTER in MetricType.
> 2) Adding a new enum GLOBAL_MUTATION_BATCH_FAILED_COUNTER in
> GlobalClientMetrics
> 3) Adding a new CombinableMetric member called mutationBatchFailed to
> MutationMetric class
> 4) Making sure that the two metrics are updated within the catch exception
> block of MutationState#send()
> 5) Unit test in PhoenixMetricsIT
> FYI, [~tdsilva]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
