[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257607#comment-15257607 ] Allen Wittenauer commented on YARN-4478: It looks to me like the YARN test is flawed, given that * the code in common doesn't break like this anywhere else that I'm aware of * we aren't doing anything special in the docker container * the actual common unit tests aren't failing If YARN doesn't work properly in a container, then that's a YARN problem. Demanding some weirdo host config is a non-starter. But again: changing the external conditions on the build servers are going to make zero difference to the config inside the container. So again, INFRA-11150 is just a waste of time. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257587#comment-15257587 ] Rohith Sharma K S commented on YARN-4478: - It got failed yesterday too!! [HadoopQA Report| https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12958048=15256199]. I am not very sure whether it is docker container setting environment problem!! > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257581#comment-15257581 ] Allen Wittenauer commented on YARN-4478: Have you seen it fail consistently in the past 10 months we've been running precommit under Docker? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257556#comment-15257556 ] Rohith Sharma K S commented on YARN-4478: - Curious to know that if precommit runs on docker, don't we face test failure like HADOOP-12687? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257550#comment-15257550 ] Allen Wittenauer commented on YARN-4478: You might as well close INFRA-11150 as won't fix given precommit runs in a docker container. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[
https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257541#comment-15257541
]
Rohith Sharma K S commented on YARN-4478:
-
Thanks [~vinodkv] for your attention on this JIRA. Yes, I will close this
Umbrella Jira since it is going very long and I think compared to where we
started this tracking, test case runs seem better.
Recently I had discussion with [~sunilg] about various test failure root causes
and made observations from this Umbrella JIRA. Some of the test case failure
which are said to be *random*, seems to be fixable. I will share some of the
observations made while fixing/reviewing/committing test cases in this umbrella
JIRA.
Types of failures seen
# Yarn event model - AsyncDispatcher : Most of the random test failures seen in
this category. For example, After registering node to RM, asserting for cluster
resource from scheduler.
{code}
rm.start();
rm.registerNode("h1:1234", 5120);
assertEquals(5120,rm.getResourceScheduler().getClusterResource());
{code}
Many a times, contributors forget while writing test cases that yarn events are
async. Many random failures are because of these events processing delay which
seems running in local eclipse tests.
# System Settings : We have seen few test case fails regularly. Mainly because
of DNS configurations. See HADOOP-12687 and INFRA-11150. This I am not sure
how should it be resolved since neither code check in preferred since it breaks
RFC standards.
# As test cases were made running in parrellel, we ran into "Address bind
exception" issues.
# MockRM APIs : In MockRM many API's are there to submit job and lunch AM which
is added over time. Few such methods are internally waiting for some events to
happen and few others explicitly need to wait for these events from test case
(contributors has to take care of this). For test case writing, contributors
should be aware of MockRM#API what it does internally. And we have seen a mix
of these apis causing random failures.
For handling open issues in this Umbrella, I will detach it and make it as Test
bug. Let us handle these separately.
> [Umbrella] : Track all the Test failures in YARN
>
>
> Key: YARN-4478
> URL: https://issues.apache.org/jira/browse/YARN-4478
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Rohith Sharma K S
>
> Recently many test cases are failing either timed out or new bug fix caused
> impact. Many test faiures JIRA are raised and are in progress.
> This is to track all the test failures JIRA's
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255884#comment-15255884 ] Sunil G commented on YARN-4478: --- [~bibinchundatt], could you please help to check this [comment|https://issues.apache.org/jira/browse/YARN-4982?focusedCommentId=15255871=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15255871] which I have given in YARN-4982. There was few points discussed on this line earlier related to DNS and we can continue the discussion in YARN-4982. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[
https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255722#comment-15255722
]
Bibin A Chundatt commented on YARN-4478:
IIUC {{new Configuration()}} is loading from classpath {{core-site.xml}} with
{{HADOOP_SECURITY_TOKEN_SERVICE_USE_IP}} value as {{false}}
{code}
public static final String HADOOP_SECURITY_TOKEN_SERVICE_USE_IP =
"hadoop.security.token.service.use_ip";
{code}
Since the value is loaded as false for hostname resolution
{{QualifiedHostResolver}} is used which is causing probelm
default value of {{hadoop.security.token.service.use_ip}} is true which should
have used {{StandardHostResolver}}. To check the same for yarn-client project
fix have uploaded patch in YARN-4982.
> [Umbrella] : Track all the Test failures in YARN
>
>
> Key: YARN-4478
> URL: https://issues.apache.org/jira/browse/YARN-4478
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Rohith Sharma K S
>
> Recently many test cases are failing either timed out or new bug fix caused
> impact. Many test faiures JIRA are raised and are in progress.
> This is to track all the test failures JIRA's
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[
https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255632#comment-15255632
]
Bibin A Chundatt commented on YARN-4478:
[~sunilg]/[~rohithsharma]
TestClientRMTokens and TestAMAuthorization looks be continuously failing still.
Will setting
{noformat}
conf.setBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP,
true);
SecurityUtil.setConfiguration(conf);
{noformat}
help in fixing the issue in test . So that in {{SecurityUtils}} will use
{{StandardHostResolver}} instead of {{QualifiedHostResolver}} the functionality
of the test will remain same. If you agree with the same can reopen the YARN
issues and fix in respective classes.
Currently yarn-api and yarn-resourcemanager classes we might have to handle.
> [Umbrella] : Track all the Test failures in YARN
>
>
> Key: YARN-4478
> URL: https://issues.apache.org/jira/browse/YARN-4478
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Rohith Sharma K S
>
> Recently many test cases are failing either timed out or new bug fix caused
> impact. Many test faiures JIRA are raised and are in progress.
> This is to track all the test failures JIRA's
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254330#comment-15254330 ] Allen Wittenauer commented on YARN-4478: Just an FYI, but there are two key problems that folks should be aware of: #1: None of the current hadoop jenkins "build project" jobs actually report all of the unit test failures in a consistent manner due to how maven works. If a dependent module fails, then the parent module is never run. In words: Given two modules A, B. Relationship is such that B requires A. If A's unit tests succeed, then B's unit tests are executed. If A's unit tests fail, then B's unit tests are *never run*. #2 I've already determined that a large chunk of the YARN unit tests CANNOT be run simultaneously due to TCP port usage. (See YARN-4950). This means that if two YARN nightlies are running on the same box at the same time, it's pretty much a 100% certainty that there will be spurious failures. (Yetus guarantees some\-\-but not total\-\-isolation via docker, so precommit should be immune to this particular problem.) That said, I've been working on a Yetus-based replacement for full compiles (YETUS-156). This would at least solve major parts of both these issues. I've been running it in test for Hadoop for a while now: (https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-test ). I've had unit tests turned off, but I just kicked off another run with the unit tests turned on so that you can see what happens. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254297#comment-15254297 ] Vinod Kumar Vavilapalli commented on YARN-4478: --- bq. Point of concern is when a QA report test failures , contributors/committers has to search for the test failures JIRA IDs and comment on their respective JIRA may be like "test failures are unrelated to this patch. test failure is tracked by YARN-" This is very paining task when there are multiple module test failures. Instead of remembering all the test failures JIRA, Umbrella JIRA would help to find easily. Actually I've tried adding more and more to this umbrella, but it is going out of hand. I kind of agree with [~kasha], we will always have failing unit tests that need fixing. Let's just use the bug-type and component from on - those are easy to search for. I'm going to use ticket-type from now on, appreciate others also doing the same. [~rohithsharma], let's use this umbrella for your current initiative and then close it down. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175002#comment-15175002 ] Rohith Sharma K S commented on YARN-4478: - bq. Also, in 20+ years of Unix system administration, I have never configured /etc/hosts with an ending period. That's because /etc/hosts resolution isn't supposed to go through the DNS resolver at all. Cool.. Then do you think original patch of HADOOP-12687 can go in? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174995#comment-15174995 ] Allen Wittenauer commented on YARN-4478: You realize that RFC is talking about DNS and not /etc/hosts, right? It's specifically to prevent the DNS resolver from adding more domains during resolution. Also, in 20+ years of Unix system administration, I have never configured /etc/hosts with an ending period. That's because /etc/hosts resolution isn't supposed to go through the DNS resolver at all. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[
https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174968#comment-15174968
]
Rohith Sharma K S commented on YARN-4478:
-
In analysis of HADOOP-12687, test failures are not because of network
connectivity which makes DNS server down.
Hadoop security model obey to RFC standards. In
[comment|https://issues.apache.org/jira/browse/HADOOP-12687?focusedCommentId=15087185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15087185],
Varun talks about RFC 1535. This RFC says hostname must be end with dot(".").
But jenkins machines does hostname are not configured with RFC which is
causing test failures.
As a test, this is able to reproduce in Ubuntu. After changing hostname ending
with dot("."), these test cases are passing. This change wants to bring in
YARN-precommit build machine too.
> [Umbrella] : Track all the Test failures in YARN
>
>
> Key: YARN-4478
> URL: https://issues.apache.org/jira/browse/YARN-4478
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Rohith Sharma K S
>
> Recently many test cases are failing either timed out or new bug fix caused
> impact. Many test faiures JIRA are raised and are in progress.
> This is to track all the test failures JIRA's
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174888#comment-15174888 ] Allen Wittenauer commented on YARN-4478: * There are plenty of examples where the Jenkins network connectivity fails, which of course would also cause DNS failures... * Changing the Jenkins servers isn't likely to fix anything given that all of the tests run in a docker container that the Hadoop project itself controls. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174874#comment-15174874 ] Rohith Sharma K S commented on YARN-4478: - Consistently few test cases are failing because of UnknowHostException. Detailed analysis given in HADOOP-12687. This is mainly because of yarn-precommit build machine hostname. I have raised INFRA JIRA INFRA-11150 to change YARN precommit build machine hostname. There were no response from INFRA team. Could any folks/PMC's knows whom to contact for resolving INFRA-11150? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065559#comment-15065559 ] Karthik Kambatla commented on YARN-4478: I am fine with using components too. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065558#comment-15065558 ] Karthik Kambatla commented on YARN-4478: We ll likely always have failing unit tests that need fixing. Should we just use a label to track these instead of an umbrella JIRA? May be create additional labels for common failure kinds - timeouts etc. for better tracking and look-up? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065611#comment-15065611 ] Rohith Sharma K S commented on YARN-4478: - I agree that currently labels OR/AND components can be named as *Test*. Point of concern is when a QA report test failures , contributors/committers has to search for the test failures JIRA IDs and comment on their respective JIRA may be like "test failures are unrelated to this patch. test failure is tracked by YARN-" This is very paining task when there are multiple module test failures. Instead of remembering all the test failures JIRA, Umbrella JIRA would help to find easily. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063472#comment-15063472 ] Sunil G commented on YARN-4478: --- Thanks for taking this initiative. I will add few test failure tickets under this which needs attention. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063489#comment-15063489 ] Rohith Sharma K S commented on YARN-4478: - Few suggestions on tracking test failures and timed out test cases # Time out tests might be a bug or may be because of test timeout is so aggressive # If these test case failures are real issue in code, then either we can covert the same jira as bug or raise new jira and link to it. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
