>>> Overall, I think the feature is important to have, but I don’t understand >>> some of the aspects of the proposal. I also don’t think there is a real >>> focus for clarity on the types of testing that are being supported here. >>> The implication is that unit tests are what this is being targeted, but is >>> this proposal specifically limiting to those particular types of tests? If >>> so, why? If not, some of the aspects really don’t make much sense to be >>> defaulted into. >> >> The proposal does not advocate any particular testing methodology. Certainly >> to start with we only support XCTest, but that is just a practical decision. >> Fundamentally the proposal is only advocating building modules that are >> *for* testing, and then executing these tests with a runner, which at first >> will be an XCTest runner. > > XCTest is just the runner with an additional style of how to validate the > test. I’m talking about the distinction between unit, integration, > performance, etc. All of these types of tests can be written with XCTest, > though some harder than others. There are choices being made with the > implicit assumption that tests in your test modules are unit tests. Running > tests by default and only running tests by default on debug builds are > examples of those implicit assumptions about the classification of test > within your modules.
We won’t be running tests by default, just building them by default. We have to pick a build-configuration as a default. I’m sorry I must be missing something, I don’t see how we are only supporting unit tests. At first XCTest is the only framework, but part of the proposal explains how in the future we will support any framework that can conform to a protocol. That protocol is the topic for another proposal. >>>> Additionally we will support directories called FooTests. This layout >>>> style is prevalent in existing open source projects and supporting it will >>>> minimize vexation for their authors. However in the interest of >>>> consistency and the corresponding reduction of cognitive-load when >>>> examining new Swift packages we will not recommend this layout. For >>>> example: >>>> >>>> Package >>>> └── Sources >>>> │ └── Foo.swift >>>> └── FooTests >>>> └── Test.swift >>> >>> Why support something that that is not going to be recommended? Prevalence >>> of something seems like a poor choice, especially when you are going to >>> specifically not recommend to use it. Also, the proposal already mentioned >>> an override mechanism to allow these to be specified. This seems like >>> something that could easily be cut. >> >> If we don’t cater to the thousands of projects already out there we >> potentially will hinder adoption. >> >> I’d agree with you if supporting this was going to hurt, but it won’t. > > It “hurts” by increasing cost, maintenance, and cognitive overload for > understanding all of the “out-of-the-box” ways that test modules get > implicitly created. True, but I don’t feel that the cost here is so great that is not a reasonable compromise relative to the other goals this project has. >>>> Additionally, we propose that building a module also builds that module's >>>> corresponding tests. Although this would result in slightly increased >>>> build times, we believe that tests are important enough to justify this >>>> (one might even consider slow building tests to be a code smell). We would >>>> prefer to go even further by executing the tests each time a module is >>>> built as well, but we understand that this would impede debug cycles. >>> >>> Re-building tests all of the time is a huge time waste. Executing those >>> tests even more so (also see original question on the types of tests being >>> supported). Not only that, in production software, it’s very often the case >>> that there are levels of tests that get run because of the shear amount of >>> them that exist and the time involved to run them. In addition to levels, >>> there are classifications of tests (perf, robustness, memory, stress, >>> fuzzing, etc…). >> >> I would like to wait and see. If you are right and it is a huge time waste >> then we can turn this off. > > Then please provide and specify how this can be opt-ed out of clearly. Is it > the --without-tests flag? Can we specific default options in the > Package.swift so that this always occurs? > > As a simple query, go around Apple and ask the compiler team if they want > this feature for their code. Ask those developing in the corelibs. Ask any of > the app teams. Please also do performance testing on this for various sized > projects. It will absolutely have an impact, especially with regards to the > current state of Swift compile times. The proposal specifies there will be a flag, the flag is TBD. Probably exactly what you specified. >>> This is something that should be an opt-in. The most basic example of this >>> is refactoring a code base (which is later briefly mentioned at the end o >>> the proposal). The first step is getting the code compiling for the >>> project. It’s not true that the very next step you take is fix up the >>> tests, especially in the cases that the refactoring/changes are exploratory. >> >> Indeed, I agree. But I’d like to wait and see. > > I really would like to understand what your metrics of acceptability are > here. Is twice the build cost ok? Three times? I’d say twice is too much. >>>> In the future, we may choose to promote the --test option to be a >>>> subcommand of the swift command itself: >>>> >>>> $ swift test >>>> >>>> However, any such decision would warrant extensive design consideration, >>>> so as to avoid polluting or crowding the command-line interface. Should >>>> there be sufficient demand and justification for it, though, it would be >>>> straightforward to add this functionality. >>> >>> >>> This doesn’t make sense to me. It’s either straightforward, or it requires >>> extensive design consideration. I personally strongly dislike coupling the >>> notion of building with test execution, so I would much rather see “swift >>> test” used, especially with a look into the future where it’s going to be >>> asked for the ability to run categories of tests and filter to a particular >>> set of tests to be run. >> >> I agree, but my issue with `swift test` is that we are beginning to tightly >> couple `swift build` with swift itself. I’m not sure we should run-and-gun >> into this decision. SwiftPM is strictly alpha/beta and any decisions we make >> on these sorts of issues can change. > > The alternatives are: > > 1. Factor out to a new binary, like spm, or > 2. Continue down the path of tightly coupling build code with test execution. > > I’d actually like to see the Swift Package Manager factor out to it’s own > entity sooner rather than later and instead shell out and use the swift > compiler. This is already how it is. > This would also help with other proposals, such as the “Getting C code > compiling” one. But that’s a different proposal... > >>> Another real problem with this type of design, is that is makes more >>> advanced build systems very complicated to make. For example, distributed >>> builds that bring together all of the compiled bits to run tests on get >>> blocked because the build command starts to expect certain intermediate >>> output. This is a real problem my previous team still has to this day with >>> xcodebuild and actively prevents us from doing this exact thing with >>> standard tools from Apple, so we have to basically roll our own. I see this >>> design following in the exact same footsteps. >> >> I’m not sure I understand. > > Every time you run “swift build” it’s going to trigger a build. This is > should never be required to run tests. Even if you say, “we’ll only re-build > on incremental builds”, you are assuming that you are building and running > tests within the same environment and that the environment is a build-capable > environment. This is not a valid assumption. This is what xcodebuild test > does today, it tries to smartly only trigger builds when necessary, but since > we build on one set of boxes and run tests on other machines, this simply > doesn’t work. > > We have the opportunity to build a proper toolset that can be correctly used > within many different workflows. However, I already see a lot of the coupling > starting to happen today that has happened within xcodebuild already, and > that really concerns me. Basically, it means that these tools aren’t going to > be usable outside of the smaller-scale apps, and teams like mine are going to > have to continue investing in more robust and flexible tools to enable our > scenarios. Well, it is not too late to uncouple these two pieces. Can you provide justification as to why we should decouple them? The engineering is not terrible to decouple them, but it would make things more complicated. Currently a good deal of the knowledge about where built products are and how they relate to targets does not need to be communicated outside of a single process, and simpler systems have less bugs, so I’d prefer not to complicate things without good rationale. >>> I think a lot of the design would be clarified by changing all of “by >>> convention” items into realized default values in the Package.swift file. >>> That would clearly demonstrate how the work and how we can change them. >> >> Can you explain what you mean? > > Show what the values for Package.swift are for this structure: > > Package > ├── Sources > │ └── Foo > │ └──Foo.swift > └── Tests > └── Foo > └── Test.swift > └── Bar > └── Test.swift > > Instead of it being “by convention”, show what actual layout of a > Package.swift file looks like to achieve the same thing. We should be able to > define a Package.swift file that does exactly the same thing. We agree. This is not part of this proposal however. But it is planned to make it possible to ignore the conventions if that is desired. The conventions are intended to speed development, but not hinder it, so when it may hinder it we intend to provide alternatives.
_______________________________________________ swift-evolution mailing list [email protected] https://lists.swift.org/mailman/listinfo/swift-evolution
