> On Jan 5, 2016, at 4:06 PM, Max Howell <max.how...@apple.com> wrote:
>
> We won’t be running tests by default, just building them by default. We have
> to pick a build-configuration as a default.
>
> I’m sorry I must be missing something, I don’t see how we are only supporting
> unit tests. At first XCTest is the only framework, but part of the proposal
> explains how in the future we will support any framework that can conform to
> a protocol. That protocol is the topic for another proposal.
When there is stuff like this in the proposal:
> "We would prefer to go even further by executing the tests each time a module
> is built as well, but we understand that this would impede debug cycles.”
That implies that there is a desire to run tests by default. That also suggests
the design of the proposal is geared for a very specific type of testing
category, specifically unit tests. Maybe that’s not what is meant, but that is
how I read it.
>> Then please provide and specify how this can be opt-ed out of clearly. Is it
>> the --without-tests flag? Can we specific default options in the
>> Package.swift so that this always occurs?
>>
>> As a simple query, go around Apple and ask the compiler team if they want
>> this feature for their code. Ask those developing in the corelibs. Ask any
>> of the app teams. Please also do performance testing on this for various
>> sized projects. It will absolutely have an impact, especially with regards
>> to the current state of Swift compile times.
>
> The proposal specifies there will be a flag, the flag is TBD. Probably
> exactly what you specified.
What about the defaults so that whenever “swift build” is run, specific flags
are used? Is that out of scope - maybe with the build configurations? Or is
that intended to be done via a layer of “build scripts” so teams can have
consistent workflows?
>>>>> In the future, we may choose to promote the --test option to be a
>>>>> subcommand of the swift command itself:
>>>>>
>>>>> $ swift test
>>>>>
>>>>> However, any such decision would warrant extensive design consideration,
>>>>> so as to avoid polluting or crowding the command-line interface. Should
>>>>> there be sufficient demand and justification for it, though, it would be
>>>>> straightforward to add this functionality.
>>>>
>>>>
>>>> This doesn’t make sense to me. It’s either straightforward, or it requires
>>>> extensive design consideration. I personally strongly dislike coupling the
>>>> notion of building with test execution, so I would much rather see “swift
>>>> test” used, especially with a look into the future where it’s going to be
>>>> asked for the ability to run categories of tests and filter to a
>>>> particular set of tests to be run.
>>>
>>> I agree, but my issue with `swift test` is that we are beginning to tightly
>>> couple `swift build` with swift itself. I’m not sure we should run-and-gun
>>> into this decision. SwiftPM is strictly alpha/beta and any decisions we
>>> make on these sorts of issues can change.
>>
>> The alternatives are:
>>
>> 1. Factor out to a new binary, like spm, or
>> 2. Continue down the path of tightly coupling build code with test execution.
>>
>> I’d actually like to see the Swift Package Manager factor out to it’s own
>> entity sooner rather than later and instead shell out and use the swift
>> compiler.
>
> This is already how it is.
Huh? Isn’t it tied to the “swift” compiler as a plug-in of sorts? Maybe I don’t
understand how it works. I’m talking about removing it completely from the
compiler infrastructure. But we can have this discussion a different time.
>> This would also help with other proposals, such as the “Getting C code
>> compiling” one. But that’s a different proposal...
>>
>>>> Another real problem with this type of design, is that is makes more
>>>> advanced build systems very complicated to make. For example, distributed
>>>> builds that bring together all of the compiled bits to run tests on get
>>>> blocked because the build command starts to expect certain intermediate
>>>> output. This is a real problem my previous team still has to this day with
>>>> xcodebuild and actively prevents us from doing this exact thing with
>>>> standard tools from Apple, so we have to basically roll our own. I see
>>>> this design following in the exact same footsteps.
>>>
>>> I’m not sure I understand.
>>
>> Every time you run “swift build” it’s going to trigger a build. This is
>> should never be required to run tests. Even if you say, “we’ll only re-build
>> on incremental builds”, you are assuming that you are building and running
>> tests within the same environment and that the environment is a
>> build-capable environment. This is not a valid assumption. This is what
>> xcodebuild test does today, it tries to smartly only trigger builds when
>> necessary, but since we build on one set of boxes and run tests on other
>> machines, this simply doesn’t work.
>>
>> We have the opportunity to build a proper toolset that can be correctly used
>> within many different workflows. However, I already see a lot of the
>> coupling starting to happen today that has happened within xcodebuild
>> already, and that really concerns me. Basically, it means that these tools
>> aren’t going to be usable outside of the smaller-scale apps, and teams like
>> mine are going to have to continue investing in more robust and flexible
>> tools to enable our scenarios.
>
> Well, it is not too late to uncouple these two pieces. Can you provide
> justification as to why we should decouple them? The engineering is not
> terrible to decouple them, but it would make things more complicated.
> Currently a good deal of the knowledge about where built products are and how
> they relate to targets does not need to be communicated outside of a single
> process, and simpler systems have less bugs, so I’d prefer not to complicate
> things without good rationale.
I don’t know what other examples to really provide here… the ability to build
is completely orthogonal to the ability to test. Given a set of binaries, I
should be able to run the tests. The tests are not always statically linked to
the testing target. Often times they are dynamically injected or used to drive
the program externally.
Maybe this provides some better context: some of our apps take multiple hours
to perform full builds. These apps are tested across multiple OS installations,
hardware configurations, and locales. The apps are not built on all of these
target machines. All of these machines should be able to run the same tests
available on a developer’s machine, and there is no fundamental reason why they
cannot be run in the same way (this is historically been the case with Xcode
projects because of arbitrary decisions and unnecessary coupling).
With this proposal, how do I run these tests without triggering a build? Is
“swift build --test” only going to run the tests without attempting to build
first? That is not how I read the proposal.
I spent a good deal of my time when I was on the Office for Mac team at
Microsoft working around all of limitations that made it extremely challenging
to actually make use of many of the tools coming out of Apple’s Xcode team. I’d
like to see the Swift tools not go down the same path and be tightly coupled in
each of the phases so that integration because a huge pain, especially when
integration needs to happen with non-Swift tools.
-David
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution
