The problem is that you have only set :requires_xcodebuild on the individual examples, and not on the example group as a whole. So when the example group is defined, there is no :requires_xcodebuild metadata, and the define_derived_metadata block does not get invoked to set :skip. Then when you call before, metadata[:skip] returns false, so it proceeds to define the hook normally.
RSpec examples inherit metadata from their parent example group (and enclosing example groups inherit from their parent group), so my suggestion is to remove :requires_xcodebuild from the individual examples in that file, and set it on the example group instead. That’ll both fix the issue and ensure that every example automatically has that metadata. Myron On Wed, Dec 20, 2017 at 6:34 AM, Jan P. <[email protected]> wrote: > That sounds perfect for my use case. As I introduced `:skip` into the > codebase, I would be pretty comfortable with adding this as well (and > confine it to non-macOS platforms to limit exposure). > > But I can't get it to work. The `before` just runs anyway: > > https://circleci.com/gh/fastlane/fastlane/12612 > https://github.com/fastlane/fastlane/blob/a103c655c4162723b96cae8f4a8b12 > 58a1419321/spec_helper.rb#L90-L98 > (Tried integrating it into my existing RSpec.configure block as well: > https://github.com/fastlane/fastlane/pull/11284/commits/ > 1815b4bca04f58ec84fd365097f61b8dd4e68b89#diff- > 82c4f19cbedb2b12a41b04cd4444aa3cR87) > > What am I missing? > > Jan > > > Am Mittwoch, 20. Dezember 2017 09:04:32 UTC+1 schrieb Myron Marston: >> >> There is a way you can simplify this further: >> >> module HookOverrides >> def before(*args) >> super unless metadata[:skip] >> endend >> RSpec.configure do |c| >> c.extend HookOverridesend >> >> This overrides before so that it’s a no-op if :skip metadata is set on >> the example group. With that in place, you don’t need to call skip from >> your before(:all) hooks. But bear in mind that if you ever set :skip on >> an example group (to set a default for the group) and then set skip: >> false for a specific example in the group…this override will cause the >> before hook to be skipped even though you’d probably expect it in that >> case. (Which is why we can’t apply this as a generic patch to RSpec itself). >> >> HTH, >> Myron >> >> >> On Tue, Dec 19, 2017 at 5:45 PM, Jan P. <[email protected]> wrote: >> >>> Thanks Myron, that worked! >>> >>> > 4704 examples, 0 failures, 175 pending >>> >>> With code: >>> >>> before(:all) do >>> skip "Requires `xcodebuild` to be installed which is not possible on >>> this platform" unless FastlaneCore::Helper.is_mac? >>> >>> As you can see right now I duplicate the reason text, and use the >>> condition that also triggers `skip`/metadata block: >>> https://github.com/fastlane/fastlane/blob/c6b1ac4621941b2efe >>> f702b923a572357c64eea9/spec_helper.rb#L68-L84 >>> >>> Is there somehow a better way to "connect" the skip in `before` to the >>> filters/metadata defined here? >>> It would be nicer if I could e.g. call a function with the "metadata" >>> ("xcodebuild") as parameter to trigger the skip or something. Any idea? >>> Best, >>> Jan >>> >>> >>> >>> Am Mittwoch, 20. Dezember 2017 02:11:56 UTC+1 schrieb Myron Marston: >>>> >>>> Unfortunately, there’s no way to use :skip metadata at the group level >>>> to skip before(:all) hooks. That’s because of how metadata in RSpec is >>>> modeled, and how RSpec implements :skip. :skip metadata is handled at >>>> the level of individual examples, and metadata is inherited from a group to >>>> its enclosing examples. So, for example, this works: >>>> >>>> RSpec.describe MyClass, :skip do >>>> it("is skipped"){ } >>>> it("is also skipped") { }end >>>> >>>> Both examples will be skipped here. >>>> >>>> But consider that you can also do this: >>>> >>>> RSpec.describe MyClass, :skip do >>>> it("is skipped"){ } >>>> it("is also skipped") { } >>>> it("is not skipped", skip: false) { }end >>>> >>>> Here we have on example that is overriding the skip: true metadata >>>> inherited from the group. As a result, a before(:all) hook can’t >>>> simply look at the group metadata to decide whether or not to skip or not. >>>> In general, before(:all) hooks have lots of gotchas like this because >>>> they don’t really fit well into the per-example semantics of so many parts >>>> of RSpec. If you can refactor your tests to no longer need such a hook >>>> (possibly using a aggregate_failures >>>> <https://relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures>), >>>> that is worth considering. Besides that, the other option you can do is to >>>> manually call the skip method from your before(:all) hook: >>>> >>>> before(:all) do >>>> skip "reason to skip" if should_skip? >>>> # rest of your hook logicend >>>> >>>> HTH, >>>> Myron >>>> >>>> >>>> >>>> >>>> On Tue, Dec 19, 2017 at 3:47 PM, Jan P. <[email protected]> wrote: >>>> >>>>> Jon, I tried >>>>> >>>>> describe "xcpretty reporter options generation", requires_xcodebuild: >>>>> true do >>>>> >>>>> >>>>> and >>>>> >>>>> before(:all), requires_xcodebuild: true do >>>>> >>>>> >>>>> and >>>>> >>>>> >>>>> describe Scan::XCPrettyReporterOptionsGenerator, requires_xcodebuild: >>>>> true do >>>>> >>>>> >>>>> but all didn't work and the tests were logged as failure because of >>>>> the `before` being executed. >>>>> >>>>> Am I doing this wrong somehow? >>>>> >>>>> >>>>> Code (last iteration) is here: >>>>> https://github.com/fastlane/fastlane/blob/janpio-mark_skippe >>>>> d_tests_as_pending_with_reason/scan/spec/xcpretty_reporter_ >>>>> options_generator_spec.rb >>>>> https://github.com/fastlane/fastlane/blob/janpio-mark_skippe >>>>> d_tests_as_pending_with_reason/spec_helper.rb#L68-L83 >>>>> Matching Circle CI run: >>>>> https://circleci.com/gh/fastlane/fastlane/12586 >>>>> >>>>> -Jan >>>>> >>>>> >>>>> >>>>> Am Dienstag, 19. Dezember 2017 23:05:53 UTC+1 schrieb Jon Rowe: >>>>>> >>>>>> You should be able to specify `requires_xcodebuild: true` on the >>>>>> describe as we’ll to prevent the before all being triggered >>>>>> >>>>>> Jon Rowe >>>>>> --------------------------- >>>>>> [email protected] >>>>>> jonrowe.co.uk >>>>>> >>>>>> On Wednesday, 20 December 2017 at 05:56, Jan P. wrote: >>>>>> >>>>>> Excuse the terrible code formatting, seems Google Groups wanted to be >>>>>> helpful. >>>>>> >>>>>> Another try: >>>>>> >>>>>> describe Scan do >>>>>> describe Scan::XCPrettyReporterOptionsGenerator do >>>>>> before(:all) do >>>>>> // code that fails when executed on non-macOS >>>>>> end >>>>>> >>>>>> describe "xcpretty reporter options generation" do >>>>>> it "generates options for the junit tempfile report required by >>>>>> scan", requires_xcodebuild: true do >>>>>> ... >>>>>> >>>>>> -J >>>>>> >>>>>> >>>>>> Am Dienstag, 19. Dezember 2017 19:55:45 UTC+1 schrieb Jan P.: >>>>>> >>>>>> Yes, that works mostly like expected: >>>>>> >>>>>> >>>>>> 4704 examples, 21 failures, 154 pending >>>>>> >>>>>> >>>>>> and >>>>>> >>>>>> ... >>>>>> [18:39:20]: ▸ Pending: (Failures listed here are expected and do not >>>>>> affect your suite's status) >>>>>> [18:39:20]: ▸ 1) Fastlane Fastlane::EnvironmentPrinter contains main >>>>>> information about the stack >>>>>> [18:39:20]: ▸ # Requires Xcode to be installed which is not possible >>>>>> on this platform >>>>>> [18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:28 >>>>>> [18:39:20]: ▸ 2) Fastlane Fastlane::EnvironmentPrinter >>>>>> FastlaneCore::Helper.xcode_version cannot be obtained contains stack >>>>>> information other than Xcode Version >>>>>> [18:39:20]: ▸ # Requires Xcode to be installed which is not possible >>>>>> on this platform >>>>>> [18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:47 >>>>>> ... >>>>>> >>>>>> >>>>>> Awesome! >>>>>> >>>>>> >>>>>> The 21 failures are new though. >>>>>> >>>>>> I have a _spec.rb file with 21 examples that has a *before(:all)* >>>>>> that seems to have been filtered with my old solution, but with *skip >>>>>> *is now executed and fails: >>>>>> >>>>>> describe Scan do >>>>>> describe Scan::XCPrettyReporterOptionsGenerator do >>>>>> before(:all) do >>>>>> .. code that fails when executed on non-macOS ... >>>>>> end >>>>>> >>>>>> describe "xcpretty reporter options generation" do >>>>>> it "generates options for the junit tempfile report required by scan", >>>>>> requires_xcodebuild: true do >>>>>> ... >>>>>> >>>>>> Any idea what I can do about this? >>>>>> -J >>>>>> >>>>>> >>>>>> >>>>>> Am Dienstag, 19. Dezember 2017 19:26:17 UTC+1 schrieb Jan P.: >>>>>> >>>>>> Thanks for the quick answer. >>>>>> >>>>>> I missed "skipping examples" because I was so happy to have found >>>>>> exclusion filters. Sounds like pretty much what I am looking for - even >>>>>> better with the explicit reason I can set for skipping. Will try and >>>>>> report >>>>>> back. >>>>>> >>>>>> Best, >>>>>> Jan >>>>>> >>>>>> >>>>>> >>>>>> Am Dienstag, 19. Dezember 2017 17:37:56 UTC+1 schrieb Myron Marston: >>>>>> >>>>>> RSpec does not provide a way to get the number of examples that were >>>>>> excluded by its inclusion or exclusion filters, but there’s a different >>>>>> mechanism that will do what you want. Instead of filtering the examples >>>>>> (which excludes them from consideration entirely), you can skip them, >>>>>> which >>>>>> prevents the body of the example from running, sets the example’s status >>>>>> to >>>>>> :pending, will print the example in yellow in the formatter output, >>>>>> and will count the example in the summary total printed at the end (e.g. >>>>>> “500 examples, 0 failures, 20 pending”). Normally, :skip metadata >>>>>> will cause an example to be skipped >>>>>> <https://www.google.com/url?q=https%3A%2F%2Frelishapp.com%2Frspec%2Frspec-core%2Fv%2F3-7%2Fdocs%2Fpending-and-skipped-examples%2Fskip-examples%23skipping-using-metadata&sa=D&sntz=1&usg=AFQjCNH2bA7au32CPocvg1H0M1Vzmcf5IQ>, >>>>>> but you’ve overwritten it to cause :skip to cause examples to be >>>>>> filtered out. >>>>>> >>>>>> Here’s my suggestion for how to wire this up. >>>>>> >>>>>> First, tag any examples that depend upon xcode with :uses_xcode >>>>>> (rather than :skip), e.g.: >>>>>> >>>>>> it "uses a feature of xcode", :xcode do >>>>>> # ...end >>>>>> >>>>>> it "does not use xcode at all" do >>>>>> # ...end >>>>>> >>>>>> Then use define_derived_metadata to automatically tag these examples >>>>>> with :skip if you are not running on OS X: >>>>>> >>>>>> # spec_helper.rbrequire 'rbconfig' >>>>>> RSpec.configure do |config| >>>>>> unless RbConfig::CONFIG['host_os'] =~ /darwin/ >>>>>> config.define_derived_metadata(:xcode) do |meta| >>>>>> meta[:skip] = "Can only be run on OS X" >>>>>> end >>>>>> endend >>>>>> >>>>>> The “Can only be run on OS X” bit will be printed in the output as >>>>>> the reason the examples are pending. >>>>>> >>>>>> HTH, >>>>>> Myron >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 19, 2017 at 3:06 AM, Jan P. <[email protected]> wrote: >>>>>> >>>>>> RSpec has this nice method to exclude individual tests/examples or >>>>>> whole groups by using filter_run_excluding in the config, then >>>>>> tagging the examples: >>>>>> >>>>>> https://relishapp.com/rspec/rspec-core/v/3-7/docs/filtering/ >>>>>> exclusion-filters >>>>>> >>>>>> RSpec.configure do |c| >>>>>> c.filter_run_excluding :skip => trueend >>>>>> RSpec.describe "something" do >>>>>> it "does one thing" do >>>>>> end >>>>>> >>>>>> it "does another thing", :skip => true do >>>>>> endend >>>>>> >>>>>> "does one thing" will be checked, >>>>>> "does another thing" will not. >>>>>> >>>>>> >>>>>> We are using this, for example, to skip some tests depending on the >>>>>> platform the test is run on by wrapping the c.filter_run_excluding >>>>>> :skip => true in an if block: >>>>>> >>>>>> If Mac, >>>>>> no exclusions, if Ubuntu, >>>>>> exclude tests that do something with Xcode. >>>>>> >>>>>> >>>>>> Right now the numbers of passing examples/test is just lower if the >>>>>> exclusion filter is used, but it would be nice to see the actual number >>>>>> of >>>>>> tests that are skipped. >>>>>> >>>>>> Is there a way to get the number of tests skipped by this method >>>>>> during a test run? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jan >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "rspec" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b- >>>>>> 9b00-ab83dfec7633%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b-9b00-ab83dfec7633%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "rspec" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051- >>>>>> a3dd-5cb942784698%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051-a3dd-5cb942784698%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "rspec" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98- >>>>> a0ee-9ebf1d9c4ae4%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98-a0ee-9ebf1d9c4ae4%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "rspec" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com >>> <https://groups.google.com/d/msgid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "rspec" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/rspec/5896f17f-84f3-48da-86f1-b99924e2a5ba%40googlegroups.com > <https://groups.google.com/d/msgid/rspec/5896f17f-84f3-48da-86f1-b99924e2a5ba%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "rspec" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/CADUxQmuJag4ZjR0UGdcb5Kr4LsnZFy5N%3Dc9hKm1uWNE0WDLjrQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
