Re: [External] : Re: NetCat Status
Hi, I will be looking after our reporting to the OpenJDK Quality Outreach on the results of testing NetBeans with new versions of Java. As has been discussed, it is NetCAT testing outcomes that could be reported. When I looked at the NetCAT page it ends at version 9 but when you select 12 the number of test categories is much shorter. I could use some guidance in interpreting the information here. This email chain expresses concern over the fact that NetCAT is so labour intensive and is not as up to date as it should be. If there is something I can contribute to help with this then let me know. Ken From: Glenn Holmer Sent: Sunday, June 13, 2021 7:54:35 PM To: netcat@netbeans.apache.org Subject: Re: [External] : Re: NetCat Status On 6/13/21 9:05 AM, Eric Bresie wrote: > (3) by demo / manual (user tested - have a user move the mouse, press a > button, select a menu, create a file, run a file, etc.). > Assuming Netcat tends towards the latter here as well. > As I understand so far, Netcat provides the specs and a means to document > things Yes, but again, the problem is that those specs haven't gotten updated for newer versions of NetBeans, and new specs haven't been written for new features of NetBeans. That's why I'm suggesting something less formal. Who is going to keep the test specs updated? If the answer is "no one", then they are useless. These test specs are non-trivial; you can see them here: https://synergy.netbeans.apache.org/#/specifications -- Glenn Holmer (Linux registered user #16682) "After the vintage season came the aftermath -- and Cenbe."
Re: [External] : Re: NetCat Status
On 6/13/21 9:05 AM, Eric Bresie wrote: (3) by demo / manual (user tested - have a user move the mouse, press a button, select a menu, create a file, run a file, etc.). Assuming Netcat tends towards the latter here as well. As I understand so far, Netcat provides the specs and a means to document things Yes, but again, the problem is that those specs haven't gotten updated for newer versions of NetBeans, and new specs haven't been written for new features of NetBeans. That's why I'm suggesting something less formal. Who is going to keep the test specs updated? If the answer is "no one", then they are useless. These test specs are non-trivial; you can see them here: https://synergy.netbeans.apache.org/#/specifications -- Glenn Holmer (Linux registered user #16682) "After the vintage season came the aftermath -- and Cenbe." - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [External] : Re: NetCat Status
My understanding of testing and coming at this with limited NetCAT experience... Testing occurs at different levels from (1) low level (unit test - do unit of functional work as expected), to (2) "middle level" (integration testing - all the components working together), to (3) regression testing (did any new changes break anything that used to work), to (4) high level (acceptance testing [1]- does it do what it said / required to do). Assuming the unit test suites get run during development and during pushes as part of github commits and/or pull request activity. Integration testing may have some coverage during the "beta testing" / "release vote" timeframe. NetCat tends to fit more in the last level with some portion of regression as well. Testing occurs in different ways (i.e. how something gets tested). It's tested by (1) automated (unit test), (2) by inspection / analysis (look at something and confirm it is as it should be - expected a calculated or outputted file has what is expected), (3) by demo / manual (user tested - have a user move the mouse, press a button, select a menu, create a file, run a file, etc.). Assuming Netcat tends towards the latter here as well. Benefits early are quicker and less manually intensive getting to possible issues earlier in development while the later more manually intensive. For the acceptance level, given its tendency to be more manual intensive and cumbersome to do this is where there is reluctance to do so more frequently. So with this in mind, is it worth more emphasis to automate more of the acceptance level testing to speed up the process and ensure the quality? As I understand so far, Netcat provides the specs and a means to document things, but it's unclear to me if this also automates (but this could be my inexperience here; does Netcat already have any automated test tools [2,3] in use yet? Maybe more things can be automated (like with JMeter [4} or cucumber (which can integrate with JIRA) [5,6] or some other tool). For the unit level of testing I believe there is good coverage here, however there may be opportunities to run some test code coverage tools [7] and find opportunities for adding coverage where there may be gaps. I know this is not necessarily an easy thing to consider (I also have a family and day job) but it may help meet the need for quality and speed up the process at the same time. Eric Bresie ebre...@gmail.com References (1) https://www.infoq.com/news/2017/04/acceptance-testing-delivery/ (2) https://www.softwaretestinghelp.com/open-source-testing-tools/ (3) https://testguild.com/automation-testing-tools/ (4) https://jmeter.apache.org/ (5) https://cucumber.io/ (6) https://cucumberforjira.atlassian.net/wiki/spaces/C4JD/overview (7) https://learning.shine.com/talenteconomy/career-help/code-coverage-tools/ On Tue, Jun 8, 2021 at 6:11 PM Scott Palmer wrote: > > > > On Jun 7, 2021, at 7:50 PM, Glenn Holmer > wrote: > > > > On 6/7/21 9:26 AM, Neil C Smith wrote: > >> On Mon, 7 Jun 2021 at 14:24, Jiří Kovalský > wrote: > >>> From that point of view it makes more sense to concentrate the testing > >>> effort around LTS version with known NetCAT start/end dates, form a > >>> dedicated team of volunteers with free capacity, certify the whole > >>> product and release it. Thanks to that we can advertise LTS as > something > >>> extra given the focused NetCAT community testing. > >>> > >>> If we would ask the same contributors to test the same stuff every > 3 > >>> months and make it an ongoing process, I think the outcome would be > lower. > >> Totally agree. If we have future LTS. I guess my comments were more > >> based on where NetCAT might evolve if LTS is dropped. Of course, an > >> annual NetCAT process might still be valid without an LTS? But it > >> does feel that the decision on whether to continue with LTS, and how > >> it works, heavily influences what NetCAT might look like in future. > > > > NetCAT was much less structured in the early days; just people choosing > an area of interest and pounding on it. Later, we had test specifications, > but there was really no one to keep them up to date and write new ones. I > think the level of detail drove potential testers away as well. > > > > Maybe NetCAT needs to be re-thought as something less formal. I think > the important thing about attracting testers is just for them to know that > the people working on the code will look into NetCAT-submitted issues more > closely, as they are coming from dedicated users. Maybe a triage system > where NetCATters try to reproduce each other's bugs? > > > > With a less structured testing program, it would be a lot easier to say > "Hey, we're getting ready for a new release! Please let's have some testing > volunteers to make sure nothing's broken!" Of course, you'd still want to > try and balance the number of people testing each area, and have some form > of recognition for the people who filed good bug reports (on the web
Re: [External] : Re: NetCat Status
> On Jun 7, 2021, at 7:50 PM, Glenn Holmer wrote: > > On 6/7/21 9:26 AM, Neil C Smith wrote: >> On Mon, 7 Jun 2021 at 14:24, Jiří Kovalský wrote: >>> From that point of view it makes more sense to concentrate the testing >>> effort around LTS version with known NetCAT start/end dates, form a >>> dedicated team of volunteers with free capacity, certify the whole >>> product and release it. Thanks to that we can advertise LTS as something >>> extra given the focused NetCAT community testing. >>> >>> If we would ask the same contributors to test the same stuff every 3 >>> months and make it an ongoing process, I think the outcome would be lower. >> Totally agree. If we have future LTS. I guess my comments were more >> based on where NetCAT might evolve if LTS is dropped. Of course, an >> annual NetCAT process might still be valid without an LTS? But it >> does feel that the decision on whether to continue with LTS, and how >> it works, heavily influences what NetCAT might look like in future. > > NetCAT was much less structured in the early days; just people choosing an > area of interest and pounding on it. Later, we had test specifications, but > there was really no one to keep them up to date and write new ones. I think > the level of detail drove potential testers away as well. > > Maybe NetCAT needs to be re-thought as something less formal. I think the > important thing about attracting testers is just for them to know that the > people working on the code will look into NetCAT-submitted issues more > closely, as they are coming from dedicated users. Maybe a triage system where > NetCATters try to reproduce each other's bugs? > > With a less structured testing program, it would be a lot easier to say "Hey, > we're getting ready for a new release! Please let's have some testing > volunteers to make sure nothing's broken!" Of course, you'd still want to try > and balance the number of people testing each area, and have some form of > recognition for the people who filed good bug reports (on the web site > release page? on social media?). And those NetBeans t-shirts were so very > cool... > > https://www.lyonlabs.org/temp/netcat-shirt.png > > -- > Glenn Holmer (Linux registered user #16682) > "After the vintage season came the aftermath -- and Cenbe.” I agree with this. I participated more in NetCat more when I wasn’t forced to join a specific group and complete a formal test plan. (…and I have the t-shirt to prove it ;-) ) It helps when I can’t be sure of how much time I will be able to allocate to NetCat as well. A forum where the collaboration can be a bit more ad hoc, with participants raising issues and others confirming the reproducibility, simply following some general guidelines and a rough idea of areas to focus on - that’s really all I would need. Regards, Scott - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [External] : Re: NetCat Status
On 6/7/21 9:26 AM, Neil C Smith wrote: On Mon, 7 Jun 2021 at 14:24, Jiří Kovalský wrote: From that point of view it makes more sense to concentrate the testing effort around LTS version with known NetCAT start/end dates, form a dedicated team of volunteers with free capacity, certify the whole product and release it. Thanks to that we can advertise LTS as something extra given the focused NetCAT community testing. If we would ask the same contributors to test the same stuff every 3 months and make it an ongoing process, I think the outcome would be lower. Totally agree. If we have future LTS. I guess my comments were more based on where NetCAT might evolve if LTS is dropped. Of course, an annual NetCAT process might still be valid without an LTS? But it does feel that the decision on whether to continue with LTS, and how it works, heavily influences what NetCAT might look like in future. NetCAT was much less structured in the early days; just people choosing an area of interest and pounding on it. Later, we had test specifications, but there was really no one to keep them up to date and write new ones. I think the level of detail drove potential testers away as well. Maybe NetCAT needs to be re-thought as something less formal. I think the important thing about attracting testers is just for them to know that the people working on the code will look into NetCAT-submitted issues more closely, as they are coming from dedicated users. Maybe a triage system where NetCATters try to reproduce each other's bugs? With a less structured testing program, it would be a lot easier to say "Hey, we're getting ready for a new release! Please let's have some testing volunteers to make sure nothing's broken!" Of course, you'd still want to try and balance the number of people testing each area, and have some form of recognition for the people who filed good bug reports (on the web site release page? on social media?). And those NetBeans t-shirts were so very cool... https://www.lyonlabs.org/temp/netcat-shirt.png -- Glenn Holmer (Linux registered user #16682) "After the vintage season came the aftermath -- and Cenbe." - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [External] : Re: NetCat Status
I also fully agree with Jirka's opinion. Not that easy to get long-term contributors. Kai On 6/7/2021 3:24 PM, Jiří Kovalský wrote: Dne 07. 06. 21 v 14:01 Neil C Smith napsal(a): [snip] Personally I think we need to figure out a way to make NetCAT a more fluid, open and ongoing thing, or drop it. Now we've moved to voted betas, rather than just advertising them, maybe we can encourage anyone testing them to pick a NetCAT test of interest to try, and report issues? In my opinion loyal long-term contributors are rare and most people in open source world prefer to help within a limited period of time between two projects/jobs/marriages and then go their own way again. From that point of view it makes more sense to concentrate the testing effort around LTS version with known NetCAT start/end dates, form a dedicated team of volunteers with free capacity, certify the whole product and release it. Thanks to that we can advertise LTS as something extra given the focused NetCAT community testing. If we would ask the same contributors to test the same stuff every 3 months and make it an ongoing process, I think the outcome would be lower. I admit I may be mistaken though so I would love to hear opinions from other NetCAT subscribers in this mailing list. Let's make this discussion objective so everyone please share your thoughts! :) -Jirka - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [External] : Re: NetCat Status
On Mon, 7 Jun 2021 at 14:24, Jiří Kovalský wrote: > From that point of view it makes more sense to concentrate the testing > effort around LTS version with known NetCAT start/end dates, form a > dedicated team of volunteers with free capacity, certify the whole > product and release it. Thanks to that we can advertise LTS as something > extra given the focused NetCAT community testing. > > If we would ask the same contributors to test the same stuff every 3 > months and make it an ongoing process, I think the outcome would be lower. Totally agree. If we have future LTS. I guess my comments were more based on where NetCAT might evolve if LTS is dropped. Of course, an annual NetCAT process might still be valid without an LTS? But it does feel that the decision on whether to continue with LTS, and how it works, heavily influences what NetCAT might look like in future. Best wishes, Neil - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [External] : Re: NetCat Status
Dne 07. 06. 21 v 14:01 Neil C Smith napsal(a): [snip] Personally I think we need to figure out a way to make NetCAT a more fluid, open and ongoing thing, or drop it. Now we've moved to voted betas, rather than just advertising them, maybe we can encourage anyone testing them to pick a NetCAT test of interest to try, and report issues? In my opinion loyal long-term contributors are rare and most people in open source world prefer to help within a limited period of time between two projects/jobs/marriages and then go their own way again. From that point of view it makes more sense to concentrate the testing effort around LTS version with known NetCAT start/end dates, form a dedicated team of volunteers with free capacity, certify the whole product and release it. Thanks to that we can advertise LTS as something extra given the focused NetCAT community testing. If we would ask the same contributors to test the same stuff every 3 months and make it an ongoing process, I think the outcome would be lower. I admit I may be mistaken though so I would love to hear opinions from other NetCAT subscribers in this mailing list. Let's make this discussion objective so everyone please share your thoughts! :) -Jirka - To unsubscribe, e-mail: netcat-unsubscr...@netbeans.apache.org For additional commands, e-mail: netcat-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
