That sounds perfect for my use case. As I introduced `:skip` into the codebase, I would be pretty comfortable with adding this as well (and confine it to non-macOS platforms to limit exposure).
But I can't get it to work. The `before` just runs anyway: https://circleci.com/gh/fastlane/fastlane/12612 https://github.com/fastlane/fastlane/blob/a103c655c4162723b96cae8f4a8b1258a1419321/spec_helper.rb#L90-L98 (Tried integrating it into my existing RSpec.configure block as well: https://github.com/fastlane/fastlane/pull/11284/commits/1815b4bca04f58ec84fd365097f61b8dd4e68b89#diff-82c4f19cbedb2b12a41b04cd4444aa3cR87) What am I missing? Jan Am Mittwoch, 20. Dezember 2017 09:04:32 UTC+1 schrieb Myron Marston: > > There is a way you can simplify this further: > > module HookOverrides > def before(*args) > super unless metadata[:skip] > endend > RSpec.configure do |c| > c.extend HookOverridesend > > This overrides before so that it’s a no-op if :skip metadata is set on > the example group. With that in place, you don’t need to call skip from > your before(:all) hooks. But bear in mind that if you ever set :skip on > an example group (to set a default for the group) and then set skip: false > for a specific example in the group…this override will cause the before > hook to be skipped even though you’d probably expect it in that case. > (Which is why we can’t apply this as a generic patch to RSpec itself). > > HTH, > Myron > > > On Tue, Dec 19, 2017 at 5:45 PM, Jan P. <[email protected] <javascript:>> > wrote: > >> Thanks Myron, that worked! >> >> > 4704 examples, 0 failures, 175 pending >> >> With code: >> >> before(:all) do >> skip "Requires `xcodebuild` to be installed which is not possible on >> this platform" unless FastlaneCore::Helper.is_mac? >> >> As you can see right now I duplicate the reason text, and use the >> condition that also triggers `skip`/metadata block: >> >> https://github.com/fastlane/fastlane/blob/c6b1ac4621941b2efef702b923a572357c64eea9/spec_helper.rb#L68-L84 >> >> Is there somehow a better way to "connect" the skip in `before` to the >> filters/metadata defined here? >> It would be nicer if I could e.g. call a function with the "metadata" >> ("xcodebuild") as parameter to trigger the skip or something. Any idea? >> Best, >> Jan >> >> >> >> Am Mittwoch, 20. Dezember 2017 02:11:56 UTC+1 schrieb Myron Marston: >>> >>> Unfortunately, there’s no way to use :skip metadata at the group level >>> to skip before(:all) hooks. That’s because of how metadata in RSpec is >>> modeled, and how RSpec implements :skip. :skip metadata is handled at >>> the level of individual examples, and metadata is inherited from a group to >>> its enclosing examples. So, for example, this works: >>> >>> RSpec.describe MyClass, :skip do >>> it("is skipped"){ } >>> it("is also skipped") { }end >>> >>> Both examples will be skipped here. >>> >>> But consider that you can also do this: >>> >>> RSpec.describe MyClass, :skip do >>> it("is skipped"){ } >>> it("is also skipped") { } >>> it("is not skipped", skip: false) { }end >>> >>> Here we have on example that is overriding the skip: true metadata >>> inherited from the group. As a result, a before(:all) hook can’t simply >>> look at the group metadata to decide whether or not to skip or not. In >>> general, before(:all) hooks have lots of gotchas like this because they >>> don’t really fit well into the per-example semantics of so many parts of >>> RSpec. If you can refactor your tests to no longer need such a hook >>> (possibly using a aggregate_failures >>> <https://relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures>), >>> >>> that is worth considering. Besides that, the other option you can do is to >>> manually call the skip method from your before(:all) hook: >>> >>> before(:all) do >>> skip "reason to skip" if should_skip? >>> # rest of your hook logicend >>> >>> HTH, >>> Myron >>> >>> >>> >>> >>> On Tue, Dec 19, 2017 at 3:47 PM, Jan P. <[email protected]> wrote: >>> >>>> Jon, I tried >>>> >>>> describe "xcpretty reporter options generation", requires_xcodebuild: true >>>> do >>>> >>>> >>>> and >>>> >>>> before(:all), requires_xcodebuild: true do >>>> >>>> >>>> and >>>> >>>> >>>> describe Scan::XCPrettyReporterOptionsGenerator, requires_xcodebuild: true >>>> do >>>> >>>> >>>> but all didn't work and the tests were logged as failure because of the >>>> `before` being executed. >>>> >>>> Am I doing this wrong somehow? >>>> >>>> >>>> Code (last iteration) is here: >>>> >>>> https://github.com/fastlane/fastlane/blob/janpio-mark_skipped_tests_as_pending_with_reason/scan/spec/xcpretty_reporter_options_generator_spec.rb >>>> >>>> https://github.com/fastlane/fastlane/blob/janpio-mark_skipped_tests_as_pending_with_reason/spec_helper.rb#L68-L83 >>>> Matching Circle CI run: >>>> https://circleci.com/gh/fastlane/fastlane/12586 >>>> >>>> -Jan >>>> >>>> >>>> >>>> Am Dienstag, 19. Dezember 2017 23:05:53 UTC+1 schrieb Jon Rowe: >>>>> >>>>> You should be able to specify `requires_xcodebuild: true` on the >>>>> describe as we’ll to prevent the before all being triggered >>>>> >>>>> Jon Rowe >>>>> --------------------------- >>>>> [email protected] >>>>> jonrowe.co.uk >>>>> >>>>> On Wednesday, 20 December 2017 at 05:56, Jan P. wrote: >>>>> >>>>> Excuse the terrible code formatting, seems Google Groups wanted to be >>>>> helpful. >>>>> >>>>> Another try: >>>>> >>>>> describe Scan do >>>>> describe Scan::XCPrettyReporterOptionsGenerator do >>>>> before(:all) do >>>>> // code that fails when executed on non-macOS >>>>> end >>>>> >>>>> describe "xcpretty reporter options generation" do >>>>> it "generates options for the junit tempfile report required by >>>>> scan", requires_xcodebuild: true do >>>>> ... >>>>> >>>>> -J >>>>> >>>>> >>>>> Am Dienstag, 19. Dezember 2017 19:55:45 UTC+1 schrieb Jan P.: >>>>> >>>>> Yes, that works mostly like expected: >>>>> >>>>> >>>>> 4704 examples, 21 failures, 154 pending >>>>> >>>>> >>>>> and >>>>> >>>>> ... >>>>> [18:39:20]: ▸ Pending: (Failures listed here are expected and do not >>>>> affect your suite's status) >>>>> [18:39:20]: ▸ 1) Fastlane Fastlane::EnvironmentPrinter contains main >>>>> information about the stack >>>>> [18:39:20]: ▸ # Requires Xcode to be installed which is not possible >>>>> on this platform >>>>> [18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:28 >>>>> [18:39:20]: ▸ 2) Fastlane Fastlane::EnvironmentPrinter >>>>> FastlaneCore::Helper.xcode_version cannot be obtained contains stack >>>>> information other than Xcode Version >>>>> [18:39:20]: ▸ # Requires Xcode to be installed which is not possible >>>>> on this platform >>>>> [18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:47 >>>>> ... >>>>> >>>>> >>>>> Awesome! >>>>> >>>>> >>>>> The 21 failures are new though. >>>>> >>>>> I have a _spec.rb file with 21 examples that has a *before(:all)* >>>>> that seems to have been filtered with my old solution, but with *skip >>>>> *is now executed and fails: >>>>> >>>>> describe Scan do >>>>> describe Scan::XCPrettyReporterOptionsGenerator do >>>>> before(:all) do >>>>> .. code that fails when executed on non-macOS ... >>>>> end >>>>> >>>>> describe "xcpretty reporter options generation" do >>>>> it "generates options for the junit tempfile report required by scan", >>>>> requires_xcodebuild: true do >>>>> ... >>>>> >>>>> Any idea what I can do about this? >>>>> -J >>>>> >>>>> >>>>> >>>>> Am Dienstag, 19. Dezember 2017 19:26:17 UTC+1 schrieb Jan P.: >>>>> >>>>> Thanks for the quick answer. >>>>> >>>>> I missed "skipping examples" because I was so happy to have found >>>>> exclusion filters. Sounds like pretty much what I am looking for - even >>>>> better with the explicit reason I can set for skipping. Will try and >>>>> report >>>>> back. >>>>> >>>>> Best, >>>>> Jan >>>>> >>>>> >>>>> >>>>> Am Dienstag, 19. Dezember 2017 17:37:56 UTC+1 schrieb Myron Marston: >>>>> >>>>> RSpec does not provide a way to get the number of examples that were >>>>> excluded by its inclusion or exclusion filters, but there’s a different >>>>> mechanism that will do what you want. Instead of filtering the examples >>>>> (which excludes them from consideration entirely), you can skip them, >>>>> which >>>>> prevents the body of the example from running, sets the example’s status >>>>> to >>>>> :pending, will print the example in yellow in the formatter output, >>>>> and will count the example in the summary total printed at the end (e.g. >>>>> “500 examples, 0 failures, 20 pending”). Normally, :skip metadata >>>>> will cause an example to be skipped >>>>> <https://www.google.com/url?q=https%3A%2F%2Frelishapp.com%2Frspec%2Frspec-core%2Fv%2F3-7%2Fdocs%2Fpending-and-skipped-examples%2Fskip-examples%23skipping-using-metadata&sa=D&sntz=1&usg=AFQjCNH2bA7au32CPocvg1H0M1Vzmcf5IQ>, >>>>> >>>>> but you’ve overwritten it to cause :skip to cause examples to be >>>>> filtered out. >>>>> >>>>> Here’s my suggestion for how to wire this up. >>>>> >>>>> First, tag any examples that depend upon xcode with :uses_xcode >>>>> (rather than :skip), e.g.: >>>>> >>>>> it "uses a feature of xcode", :xcode do >>>>> # ...end >>>>> >>>>> it "does not use xcode at all" do >>>>> # ...end >>>>> >>>>> Then use define_derived_metadata to automatically tag these examples >>>>> with :skip if you are not running on OS X: >>>>> >>>>> # spec_helper.rbrequire 'rbconfig' >>>>> RSpec.configure do |config| >>>>> unless RbConfig::CONFIG['host_os'] =~ /darwin/ >>>>> config.define_derived_metadata(:xcode) do |meta| >>>>> meta[:skip] = "Can only be run on OS X" >>>>> end >>>>> endend >>>>> >>>>> The “Can only be run on OS X” bit will be printed in the output as the >>>>> reason the examples are pending. >>>>> >>>>> HTH, >>>>> Myron >>>>> >>>>> >>>>> >>>>> On Tue, Dec 19, 2017 at 3:06 AM, Jan P. <[email protected]> wrote: >>>>> >>>>> RSpec has this nice method to exclude individual tests/examples or >>>>> whole groups by using filter_run_excluding in the config, then >>>>> tagging the examples: >>>>> >>>>> >>>>> https://relishapp.com/rspec/rspec-core/v/3-7/docs/filtering/exclusion-filters >>>>> >>>>> RSpec.configure do |c| >>>>> c.filter_run_excluding :skip => trueend >>>>> RSpec.describe "something" do >>>>> it "does one thing" do >>>>> end >>>>> >>>>> it "does another thing", :skip => true do >>>>> endend >>>>> >>>>> "does one thing" will be checked, >>>>> "does another thing" will not. >>>>> >>>>> >>>>> We are using this, for example, to skip some tests depending on the >>>>> platform the test is run on by wrapping the c.filter_run_excluding >>>>> :skip => true in an if block: >>>>> >>>>> If Mac, >>>>> no exclusions, if Ubuntu, >>>>> exclude tests that do something with Xcode. >>>>> >>>>> >>>>> Right now the numbers of passing examples/test is just lower if the >>>>> exclusion filter is used, but it would be nice to see the actual number >>>>> of >>>>> tests that are skipped. >>>>> >>>>> Is there a way to get the number of tests skipped by this method >>>>> during a test run? >>>>> >>>>> Thanks, >>>>> >>>>> Jan >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "rspec" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b-9b00-ab83dfec7633%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b-9b00-ab83dfec7633%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "rspec" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051-a3dd-5cb942784698%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051-a3dd-5cb942784698%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "rspec" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98-a0ee-9ebf1d9c4ae4%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98-a0ee-9ebf1d9c4ae4%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "rspec" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] <javascript:> >> . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com >> >> <https://groups.google.com/d/msgid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "rspec" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/5896f17f-84f3-48da-86f1-b99924e2a5ba%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
