On Fri, Jun 29, 2018 at 4:38 AM, Mike Blumenkrantz < michael.blumenkra...@gmail.com> wrote:
> I agree that app testers are good, but ideally unit tests should cover all > the codepaths which can be triggered by an app. We can use tools like lcov > to verify that the same codepaths are being taken and ensure that identical > code is being run in tests as in apps. > > Testing in a controlled environment will always be better than random app > testing by users, provided that the coverage of the tests is good enough. > How many tickets have we had which are not able to be reproduced in an app > by a developer after being reported by a user? This is the purpose of good > unit testing: to eliminate that gap in coverage by guaranteeing the same > code is run every time. > > With this said, our unit tests are not currently good enough to provide > this level of coverage, so adding app testing will be useful for the > foreseeable future until we can achieve a significantly better degree of > coverage. > > Targeting the beta release as the point when all compatibility issues are > resolved seems reasonable to me; we have never had clear thresholds for > what determines each release, so perhaps we can use something like: > > Alpha - all "urgent" tickets resolved, excluding regressions > Beta - all compatibility issues (regressions) resolved > > After this point we can reassess the release state. > > If we are going to explicitly include app testing in the release testing > process then I think it's important to have a list of apps which are tested > along with test cases for those apps. I don't personally use any EFL-based > apps in my daily life, so just opening an app and clicking around randomly > will probably not be as effective as a more focused plan. > My point here is, We can firstly ask to the actual app maintainers. Otherwise, we can resign designated testers for each apps. Non-maintained apps + non useful apps, we will ignore them in the end, But at least we can consider apps, get a consensus with app maintainers(app testers) to confirm ready to release efl or should fix it more. I think this is a practical way to improve our efl eco system. > > On Wed, Jun 27, 2018 at 10:08 PM Hermet Park <hermetp...@gmail.com> wrote: > > > Additional explanation for the second, > > > > Unit test, TC, and automation tool is good but not enough. > > We need human testers absolutely. They could test apps, detect errors and > > then report it. > > For the progressive sw, this must be a mandatory, not an optional > process. > > > > For version release(maybe in the alpha step or just before alpha), > > We officially announce about app test and list up candidate apps for the > > test. > > > > Basically each app maintainers(actual app developers) could take cover > > their apps. > > In case of absence of maintainer(but still the app is good to maintain), > we > > can delegate a tester for the app. > > Even if 1 app, 1 tester is good. People who are familiar with the app is > > better. > > > > For the delegated case, testers can try all possible functionalities of > the > > app. If they find any wrong, weird behaviors, > > they can compare the app working on the previous efl version. > > > > If a wrong behavior is only showed on new efl version, they can list up > the > > issue and share with community officially. > > > > If efl developers approve the issue is about compatibilty issue, they > must > > fix it before beta. > > > > > > On Wed, Jun 27, 2018 at 9:18 PM, Mike Blumenkrantz < > > michael.blumenkra...@gmail.com> wrote: > > > > > It seems like you have two main points here: > > > > > > 1) The review process for components should require the approval of > > > "maintainers", or people who are experts in a given area, and not just > > > anyone with commit access. > > > > > > I agree with this idea. While it would certainly be nice if everyone > > could > > > review every patch, not everyone has the same level of expertise in > every > > > part of the codebase, and having someone without enough knowledge of > the > > > code doing review on it can lead to problems. > > > I think the next step for this would be to create a proposal with some > > > initial "maintainers" for given parts of the codebase and set up herald > > > rules to automatically add them as reviewers when patches are submitted > > for > > > their areas. Then we can see which areas need more coverage? > > > > > > > > > 2) Testing is still inadequate, and applications need to work with > > versions > > > of EFL that are about to ship. > > > > > > This is also a good point. There are cases where applications are > broken > > by > > > changes in EFL, and we should avoid this. The problem, however, is that > > > application-based testing will always be a bit random due to > differences > > in > > > user configs and distro setups. > > > We should resolve this by continuing to improve our unit testing as > well > > as > > > working to integrate Exactness into the test process. As coverage of > > these > > > two types of testing increase, the difference between "testing by > running > > > an app" and "testing by running 'make check'" will lessen until > > eventually > > > only 'make check' should be needed. This also requires some work from > app > > > developers: any time their app is broken, they should ensure there is a > > > clear report, and a unit test of some kind should be added to ensure > that > > > the issue does not occur again. > > > > > > On Mon, Jun 25, 2018 at 11:54 PM Hermet Park <hermetp...@gmail.com> > > wrote: > > > > > > > Thankfully, Xavi Artigas fixed. Even though unstable but it's > > compilable > > > on > > > > devel branch now. > > > > > > > > Actually, efl has been bad at compatibility so far. > > > > After 1 year leaving, Enventor has totally collapsed which was stable > > > > enough. > > > > It's not about interface nor compilation but behaviors. > > > > UIs is too fragile now, entry and focus in Enventor showed me the > worst > > > > compatibility among that. > > > > It's too shame. > > > > > > > > Since efl is a very huge system. many committers are not easy to > > > understand > > > > whole bunch of code and history. > > > > > > > > I agree with zmike, bu5hm4n and other people about review process. We > > > need > > > > more strict patch review. > > > > But I'd rather say, it doesn't mean just review but the real review > by > > > > actual maintainers. > > > > Submitting patches without actual maintainer's review shoudln't be > > > > acceptable even if they are committer. > > > > If a non-maintainer reviews code, verifies it, situation won't be > > > > different. > > > > > > > > In my experience, unstable efl has not come from the misunderstanding > > of > > > > programming language, eye-catching compatibility breakage, crash and > > > simple > > > > logic change inside the function. > > > > Sometimes we do mistake but it's not too critical. We can easily find > > > them > > > > and fix them soon by compiler, code analizyer, test apps and > > developers. > > > > > > > > What's on the stake? > > > > > > > > Unstable efl has coming from the misunderstanding of code history, > > whole > > > > logic sequence in real complex usage and api concept changes. > > > > > > > > Obviously, we need official designated maintainers in efl, at least > > those > > > > maintaining parts should not be broken any more. > > > > > > > > Plus, as we know, UI is not a single functional stuff. It's > applicable. > > > > unit test or single function test is not enough. > > > > Even we human cannot imagine or expect how our changes will bring > > broken > > > > behaviors from this complex world. > > > > > > > > few but still, we have those efl apps and active app maintainers- > rage, > > > > terminology, eruler, ephoto and enventor etc > > > > Actually they are not enough number of unit but unless we try to keep > > > them > > > > alive, well, who wanna untrustworthy library and write apps using it? > > > > > > > > I wish efl developers should not satisfy with just unit test / test > > apps > > > > such as elementary_test. > > > > Before a new version release, additionally, we should get approval by > > > more > > > > practical apps whether they have any compatibility issues or not. > > > > > > > > > > > > On Tue, Jun 26, 2018 at 11:11 AM, Simon Lees <sfl...@suse.de> wrote: > > > > > > > > > > > > > > > > > > > On 26/06/18 04:36, Mike Blumenkrantz wrote: > > > > > > Hi, > > > > > > > > > > > > Do not attempt to use Eflete. > > > > > > > > > > > > I was attempting to examine a related issue from phab today on my > > > test > > > > > > machine and Eflete somehow managed to delete nearly EVERYTHING > from > > > my > > > > > home > > > > > > directory. All my .directories, all my source trees, everything > > > > deleted. > > > > > > > > > > > > Fortunately, this was only a testing machine and nothing other > than > > > > some > > > > > > local configs were lost. > > > > > > > > > > > > DO NOT USE EFLETE. > > > > > > > > > > > > > > > > > > Regards, > > > > > > Mike > > > > > > > > > > Last time I tried to use it, it wouldn't build against the current > > > > > stable efl release so I gave up, which is a shame because when it > was > > > > > working well it was a really useful tool. > > > > > > > > > > -- > > > > > > > > > > Simon Lees (Simotek) http://simotek.net > > > > > > > > > > Emergency Update Team keybase.io/simotek > > > > > SUSE Linux Adelaide Australia, UTC+10:30 > > > > > GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > ------------------ > > > > > Check out the vibrant tech community on one of the world's most > > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > > _______________________________________________ > > > > > enlightenment-devel mailing list > > > > > enlightenment-devel@lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, Hermet > > > > > > > > ------------------------------------------------------------ > > > ------------------ > > > > Check out the vibrant tech community on one of the world's most > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > > > > enlightenment-devel mailing list > > > > enlightenment-devel@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > > > ------------------------------------------------------------ > > > ------------------ > > > Check out the vibrant tech community on one of the world's most > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > enlightenment-devel mailing list > > > enlightenment-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > > > > > > > -- > > Regards, Hermet > > > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > enlightenment-devel mailing list > > enlightenment-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > enlightenment-devel mailing list > enlightenment-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > -- Regards, Hermet ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
