Thanks for the explanation, now it makes sense what was happening. I applied that to our project and it is working as expected. Thank you so much for your help.
Can a `before` hook ever access the metadata of an example (`before(:each)`?), or does this only get set after `before` is already finished executing? I am lucky to not have a test file right now where I have to differentiate between tests with a `before` that should run and tests that shold not - but I am sure this will happen sooner or later :/ Jan Am Mittwoch, 20. Dezember 2017 17:30:11 UTC+1 schrieb Myron Marston: > > The problem is that you have only set :requires_xcodebuild on the > individual examples, and not on the example group as a whole. So when the > example group is defined, there is no :requires_xcodebuild metadata, and > the define_derived_metadata block does not get invoked to set :skip. Then > when you call before, metadata[:skip] returns false, so it proceeds to > define the hook normally. > > RSpec examples inherit metadata from their parent example group (and > enclosing example groups inherit from their parent group), so my suggestion > is to remove :requires_xcodebuild from the individual examples in that > file, and set it on the example group instead. That’ll both fix the issue > and ensure that every example automatically has that metadata. > > Myron > > > On Wed, Dec 20, 2017 at 6:34 AM, Jan P. <[email protected] <javascript:>> > wrote: > >> That sounds perfect for my use case. As I introduced `:skip` into the >> codebase, I would be pretty comfortable with adding this as well (and >> confine it to non-macOS platforms to limit exposure). >> >> But I can't get it to work. The `before` just runs anyway: >> >> https://circleci.com/gh/fastlane/fastlane/12612 >> >> https://github.com/fastlane/fastlane/blob/a103c655c4162723b96cae8f4a8b1258a1419321/spec_helper.rb#L90-L98 >> (Tried integrating it into my existing RSpec.configure block as well: >> https://github.com/fastlane/fastlane/pull/11284/commits/1815b4bca04f58ec84fd365097f61b8dd4e68b89#diff-82c4f19cbedb2b12a41b04cd4444aa3cR87 >> ) >> >> What am I missing? >> >> Jan >> >> >> Am Mittwoch, 20. Dezember 2017 09:04:32 UTC+1 schrieb Myron Marston: >>> >>> There is a way you can simplify this further: >>> >>> module HookOverrides >>> def before(*args) >>> super unless metadata[:skip] >>> endend >>> RSpec.configure do |c| >>> c.extend HookOverridesend >>> >>> This overrides before so that it’s a no-op if :skip metadata is set on >>> the example group. With that in place, you don’t need to call skip from >>> your before(:all) hooks. But bear in mind that if you ever set :skip on >>> an example group (to set a default for the group) and then set skip: >>> false for a specific example in the group…this override will cause the >>> before hook to be skipped even though you’d probably expect it in that >>> case. (Which is why we can’t apply this as a generic patch to RSpec itself). >>> >>> HTH, >>> Myron >>> >>> >>> On Tue, Dec 19, 2017 at 5:45 PM, Jan P. <[email protected]> wrote: >>> >>>> Thanks Myron, that worked! >>>> >>>> > 4704 examples, 0 failures, 175 pending >>>> >>>> With code: >>>> >>>> before(:all) do >>>> skip "Requires `xcodebuild` to be installed which is not possible on >>>> this platform" unless FastlaneCore::Helper.is_mac? >>>> >>>> As you can see right now I duplicate the reason text, and use the >>>> condition that also triggers `skip`/metadata block: >>>> >>>> https://github.com/fastlane/fastlane/blob/c6b1ac4621941b2efef702b923a572357c64eea9/spec_helper.rb#L68-L84 >>>> >>>> Is there somehow a better way to "connect" the skip in `before` to the >>>> filters/metadata defined here? >>>> It would be nicer if I could e.g. call a function with the "metadata" >>>> ("xcodebuild") as parameter to trigger the skip or something. Any idea? >>>> Best, >>>> Jan >>>> >>>> >>>> >>>> Am Mittwoch, 20. Dezember 2017 02:11:56 UTC+1 schrieb Myron Marston: >>>>> >>>>> Unfortunately, there’s no way to use :skip metadata at the group >>>>> level to skip before(:all) hooks. That’s because of how metadata in >>>>> RSpec is modeled, and how RSpec implements :skip. :skip metadata is >>>>> handled at the level of individual examples, and metadata is inherited >>>>> from >>>>> a group to its enclosing examples. So, for example, this works: >>>>> >>>>> RSpec.describe MyClass, :skip do >>>>> it("is skipped"){ } >>>>> it("is also skipped") { }end >>>>> >>>>> Both examples will be skipped here. >>>>> >>>>> But consider that you can also do this: >>>>> >>>>> RSpec.describe MyClass, :skip do >>>>> it("is skipped"){ } >>>>> it("is also skipped") { } >>>>> it("is not skipped", skip: false) { }end >>>>> >>>>> Here we have on example that is overriding the skip: true metadata >>>>> inherited from the group. As a result, a before(:all) hook can’t >>>>> simply look at the group metadata to decide whether or not to skip or >>>>> not. >>>>> In general, before(:all) hooks have lots of gotchas like this because >>>>> they don’t really fit well into the per-example semantics of so many >>>>> parts >>>>> of RSpec. If you can refactor your tests to no longer need such a hook >>>>> (possibly using a aggregate_failures >>>>> <https://relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures>), >>>>> >>>>> that is worth considering. Besides that, the other option you can do is >>>>> to >>>>> manually call the skip method from your before(:all) hook: >>>>> >>>>> before(:all) do >>>>> skip "reason to skip" if should_skip? >>>>> # rest of your hook logicend >>>>> >>>>> HTH, >>>>> Myron >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Dec 19, 2017 at 3:47 PM, Jan P. <[email protected]> wrote: >>>>> >>>>>> Jon, I tried >>>>>> >>>>>> describe "xcpretty reporter options generation", requires_xcodebuild: >>>>>> true do >>>>>> >>>>>> >>>>>> and >>>>>> >>>>>> before(:all), requires_xcodebuild: true do >>>>>> >>>>>> >>>>>> and >>>>>> >>>>>> >>>>>> describe Scan::XCPrettyReporterOptionsGenerator, requires_xcodebuild: >>>>>> true do >>>>>> >>>>>> >>>>>> but all didn't work and the tests were logged as failure because of >>>>>> the `before` being executed. >>>>>> >>>>>> Am I doing this wrong somehow? >>>>>> >>>>>> >>>>>> Code (last iteration) is here: >>>>>> >>>>>> https://github.com/fastlane/fastlane/blob/janpio-mark_skipped_tests_as_pending_with_reason/scan/spec/xcpretty_reporter_options_generator_spec.rb >>>>>> >>>>>> https://github.com/fastlane/fastlane/blob/janpio-mark_skipped_tests_as_pending_with_reason/spec_helper.rb#L68-L83 >>>>>> Matching Circle CI run: >>>>>> https://circleci.com/gh/fastlane/fastlane/12586 >>>>>> >>>>>> -Jan >>>>>> >>>>>> >>>>>> >>>>>> Am Dienstag, 19. Dezember 2017 23:05:53 UTC+1 schrieb Jon Rowe: >>>>>>> >>>>>>> You should be able to specify `requires_xcodebuild: true` on the >>>>>>> describe as we’ll to prevent the before all being triggered >>>>>>> >>>>>>> Jon Rowe >>>>>>> --------------------------- >>>>>>> [email protected] >>>>>>> jonrowe.co.uk >>>>>>> >>>>>>> On Wednesday, 20 December 2017 at 05:56, Jan P. wrote: >>>>>>> >>>>>>> Excuse the terrible code formatting, seems Google Groups wanted to >>>>>>> be helpful. >>>>>>> >>>>>>> Another try: >>>>>>> >>>>>>> describe Scan do >>>>>>> describe Scan::XCPrettyReporterOptionsGenerator do >>>>>>> before(:all) do >>>>>>> // code that fails when executed on non-macOS >>>>>>> end >>>>>>> >>>>>>> describe "xcpretty reporter options generation" do >>>>>>> it "generates options for the junit tempfile report required by >>>>>>> scan", requires_xcodebuild: true do >>>>>>> ... >>>>>>> >>>>>>> -J >>>>>>> >>>>>>> >>>>>>> Am Dienstag, 19. Dezember 2017 19:55:45 UTC+1 schrieb Jan P.: >>>>>>> >>>>>>> Yes, that works mostly like expected: >>>>>>> >>>>>>> >>>>>>> 4704 examples, 21 failures, 154 pending >>>>>>> >>>>>>> >>>>>>> and >>>>>>> >>>>>>> ... >>>>>>> [18:39:20]: ▸ Pending: (Failures listed here are expected and do not >>>>>>> affect your suite's status) >>>>>>> [18:39:20]: ▸ 1) Fastlane Fastlane::EnvironmentPrinter contains main >>>>>>> information about the stack >>>>>>> [18:39:20]: ▸ # Requires Xcode to be installed which is not possible >>>>>>> on this platform >>>>>>> [18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:28 >>>>>>> [18:39:20]: ▸ 2) Fastlane Fastlane::EnvironmentPrinter >>>>>>> FastlaneCore::Helper.xcode_version cannot be obtained contains stack >>>>>>> information other than Xcode Version >>>>>>> [18:39:20]: ▸ # Requires Xcode to be installed which is not possible >>>>>>> on this platform >>>>>>> [18:39:20]: ▸ # ./fastlane/spec/env_spec.rb:47 >>>>>>> ... >>>>>>> >>>>>>> >>>>>>> Awesome! >>>>>>> >>>>>>> >>>>>>> The 21 failures are new though. >>>>>>> >>>>>>> I have a _spec.rb file with 21 examples that has a *before(:all)* >>>>>>> that seems to have been filtered with my old solution, but with *skip >>>>>>> *is now executed and fails: >>>>>>> >>>>>>> describe Scan do >>>>>>> describe Scan::XCPrettyReporterOptionsGenerator do >>>>>>> before(:all) do >>>>>>> .. code that fails when executed on non-macOS ... >>>>>>> end >>>>>>> >>>>>>> describe "xcpretty reporter options generation" do >>>>>>> it "generates options for the junit tempfile report required by scan", >>>>>>> requires_xcodebuild: true do >>>>>>> ... >>>>>>> >>>>>>> Any idea what I can do about this? >>>>>>> -J >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am Dienstag, 19. Dezember 2017 19:26:17 UTC+1 schrieb Jan P.: >>>>>>> >>>>>>> Thanks for the quick answer. >>>>>>> >>>>>>> I missed "skipping examples" because I was so happy to have found >>>>>>> exclusion filters. Sounds like pretty much what I am looking for - even >>>>>>> better with the explicit reason I can set for skipping. Will try and >>>>>>> report >>>>>>> back. >>>>>>> >>>>>>> Best, >>>>>>> Jan >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am Dienstag, 19. Dezember 2017 17:37:56 UTC+1 schrieb Myron Marston: >>>>>>> >>>>>>> RSpec does not provide a way to get the number of examples that were >>>>>>> excluded by its inclusion or exclusion filters, but there’s a different >>>>>>> mechanism that will do what you want. Instead of filtering the examples >>>>>>> (which excludes them from consideration entirely), you can skip them, >>>>>>> which >>>>>>> prevents the body of the example from running, sets the example’s >>>>>>> status to >>>>>>> :pending, will print the example in yellow in the formatter output, >>>>>>> and will count the example in the summary total printed at the end >>>>>>> (e.g. >>>>>>> “500 examples, 0 failures, 20 pending”). Normally, :skip metadata >>>>>>> will cause an example to be skipped >>>>>>> <https://www.google.com/url?q=https%3A%2F%2Frelishapp.com%2Frspec%2Frspec-core%2Fv%2F3-7%2Fdocs%2Fpending-and-skipped-examples%2Fskip-examples%23skipping-using-metadata&sa=D&sntz=1&usg=AFQjCNH2bA7au32CPocvg1H0M1Vzmcf5IQ>, >>>>>>> >>>>>>> but you’ve overwritten it to cause :skip to cause examples to be >>>>>>> filtered out. >>>>>>> >>>>>>> Here’s my suggestion for how to wire this up. >>>>>>> >>>>>>> First, tag any examples that depend upon xcode with :uses_xcode >>>>>>> (rather than :skip), e.g.: >>>>>>> >>>>>>> it "uses a feature of xcode", :xcode do >>>>>>> # ...end >>>>>>> >>>>>>> it "does not use xcode at all" do >>>>>>> # ...end >>>>>>> >>>>>>> Then use define_derived_metadata to automatically tag these >>>>>>> examples with :skip if you are not running on OS X: >>>>>>> >>>>>>> # spec_helper.rbrequire 'rbconfig' >>>>>>> RSpec.configure do |config| >>>>>>> unless RbConfig::CONFIG['host_os'] =~ /darwin/ >>>>>>> config.define_derived_metadata(:xcode) do |meta| >>>>>>> meta[:skip] = "Can only be run on OS X" >>>>>>> end >>>>>>> endend >>>>>>> >>>>>>> The “Can only be run on OS X” bit will be printed in the output as >>>>>>> the reason the examples are pending. >>>>>>> >>>>>>> HTH, >>>>>>> Myron >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Dec 19, 2017 at 3:06 AM, Jan P. <[email protected]> wrote: >>>>>>> >>>>>>> RSpec has this nice method to exclude individual tests/examples or >>>>>>> whole groups by using filter_run_excluding in the config, then >>>>>>> tagging the examples: >>>>>>> >>>>>>> >>>>>>> https://relishapp.com/rspec/rspec-core/v/3-7/docs/filtering/exclusion-filters >>>>>>> >>>>>>> RSpec.configure do |c| >>>>>>> c.filter_run_excluding :skip => trueend >>>>>>> RSpec.describe "something" do >>>>>>> it "does one thing" do >>>>>>> end >>>>>>> >>>>>>> it "does another thing", :skip => true do >>>>>>> endend >>>>>>> >>>>>>> "does one thing" will be checked, >>>>>>> "does another thing" will not. >>>>>>> >>>>>>> >>>>>>> We are using this, for example, to skip some tests depending on the >>>>>>> platform the test is run on by wrapping the c.filter_run_excluding >>>>>>> :skip => true in an if block: >>>>>>> >>>>>>> If Mac, >>>>>>> no exclusions, if Ubuntu, >>>>>>> exclude tests that do something with Xcode. >>>>>>> >>>>>>> >>>>>>> Right now the numbers of passing examples/test is just lower if the >>>>>>> exclusion filter is used, but it would be nice to see the actual number >>>>>>> of >>>>>>> tests that are skipped. >>>>>>> >>>>>>> Is there a way to get the number of tests skipped by this method >>>>>>> during a test run? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jan >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "rspec" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b-9b00-ab83dfec7633%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/rspec/3108ef8e-303d-425b-9b00-ab83dfec7633%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "rspec" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051-a3dd-5cb942784698%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/rspec/bcfd5676-92bd-4051-a3dd-5cb942784698%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "rspec" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98-a0ee-9ebf1d9c4ae4%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/rspec/715c3e8b-c8e0-4c98-a0ee-9ebf1d9c4ae4%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "rspec" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/rspec/bb687ad6-5489-4342-b1c8-c1656b9bf2f3%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "rspec" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] <javascript:> >> . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/rspec/5896f17f-84f3-48da-86f1-b99924e2a5ba%40googlegroups.com >> >> <https://groups.google.com/d/msgid/rspec/5896f17f-84f3-48da-86f1-b99924e2a5ba%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "rspec" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/f001d009-28cb-4be2-9c41-2f8fa29893a1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
