I agree with Robert. Also there is usually a workaround to unbrake previous version by installing a specific version of an offending dependency.
On Fri, Dec 15, 2017 at 2:53 PM, Robert Bradshaw <[email protected]> wrote: > This has the downside of pinning the dependencies for all downstream > projects, making it impossible for them to use different versions than > the ones we happened to choose. (Imagine the pain of two or more of > our dependencies pinned all their dependencies...) > > On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <[email protected]> wrote: > > +1 to pinning to exact versions, to be sure that our releases do not > break > > when newer versions of dependencies are released. > > > > On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <[email protected]> wrote: > >> > >> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath < > [email protected]> > >> wrote: > >>> > >>> +1 for automating the process of checking for possible version bumps. > >>> > >>> Also, what do you think about pinning dependencies to exact versions > >>> (instead of ranges) after cutting a release branch ? This should > improve the > >>> stability of released SDKs (but not a prefect solution since transitive > >>> dependencies can still change). > >> > >> > >> This is a reasonable suggestion. The issue with that is, by being less > >> flexible we will prevent users from using latest versions of > dependencies. > >> On the other hand it will prevent breaking of already released versions. > >> > >>> > >>> > >>> Thanks, > >>> Cham > >>> > >>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <[email protected]> wrote: > >>>> > >>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <[email protected] > > > >>>> wrote: > >>>>> > >>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <[email protected]> > wrote: > >>>>> > > >>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw > >>>>> > <[email protected]> > >>>>> > wrote: > >>>>> >> > >>>>> >> I am also in favor of pinning as an immediate fix, bumping the > bound > >>>>> >> otherwise. > >>>>> >> > >>>>> >> Regarding putting an upper bound to avoid being broken, the last > two > >>>>> >> breaks have been due to just having an (unneeded) upper bound > (which > >>>>> >> held us back to broken/incompatible releases in relationship to > >>>>> >> other > >>>>> >> dependencies). We should try to trust semantic versioning when > >>>>> >> possible, and when not we must regularly audit. > >>>>> > > >>>>> > +1 to this, especially the auditing part. We also had breaks > because > >>>>> > we > >>>>> > trusted semantic versioning. So far our semi-official policy was to > >>>>> > trust a > >>>>> > package until they prove it otherwise. I will argue that grpc here > is > >>>>> > making > >>>>> > a breaking change in a minor version increment by changing the way > >>>>> > they are > >>>>> > depending on a major package. > >>>>> > >>>>> A minor version bump should be allowed to require a minor version > bump > >>>>> in its dependencies. > >>>>> > >>>>> > We have done a good job of auditing and updating those pinned (or > >>>>> > upper > >>>>> > bounded) dependencies, and probably we are behind in some of those. > >>>>> > > >>>>> > I wonder if we can automate some of this? If we can get a report, > >>>>> > that > >>>>> > audits our dependencies, warns us about new releases and potential > >>>>> > conflicts > >>>>> > it would be much easier to keep things up to date. > >>>>> > >>>>> Big +1, it should be easy to set up a nightly that relaxes some of > the > >>>>> requirements and sees what (if anything) breaks. Not breaking is > >>>>> likely a signal that we should relax ours. > >>>> > >>>> > >>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. > I > >>>> think it would be awesome if we can tackle this as part of a better > >>>> infrastructure for testing work. > >>>> > >>>>> > >>>>> > >>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA) > >>>>> >> <[email protected]> wrote: > >>>>> >> > > >>>>> >> > [ > >>>>> >> > > >>>>> >> > https://issues.apache.org/jira/browse/BEAM-3357?page= > com.atlassian.jira.plugin.system.issuetabpanels:comment- > tabpanel&focusedCommentId=16293276#comment-16293276 > >>>>> >> > ] > >>>>> >> > > >>>>> >> > Chamikara Jayalath commented on BEAM-3357: > >>>>> >> > ------------------------------------------ > >>>>> >> > > >>>>> >> > You mean we could bump up the upper bound ? I think we should > keep > >>>>> >> > some > >>>>> >> > upper bound in case we get badly broken by a future protobuf > >>>>> >> > release. > >>>>> >> > > >>>>> >> >> Python SDK head fails to run tests due to > >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0') > >>>>> >> >> > >>>>> >> >> > >>>>> >> >> ------------------------------------------------------------ > -------------------------- > >>>>> >> >> > >>>>> >> >> Key: BEAM-3357 > >>>>> >> >> URL: > >>>>> >> >> https://issues.apache.org/jira/browse/BEAM-3357 > >>>>> >> >> Project: Beam > >>>>> >> >> Issue Type: Bug > >>>>> >> >> Components: sdk-py-core > >>>>> >> >> Reporter: Chamikara Jayalath > >>>>> >> >> Priority: Critical > >>>>> >> >> > >>>>> >> >> Error is: > >>>>> >> >> running build_ext > >>>>> >> >> Traceback (most recent call last): > >>>>> >> >> File "setup.py", line 202, in <module> > >>>>> >> >> 'test': generate_protos_first(test), > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ > proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py", > >>>>> >> >> line 129, in setup > >>>>> >> >> return distutils.core.setup(**attrs) > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ > lib/python2.7/distutils/core.py", > >>>>> >> >> line 151, in setup > >>>>> >> >> dist.run_commands() > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ > lib/python2.7/distutils/dist.py", > >>>>> >> >> line 953, in run_commands > >>>>> >> >> self.run_command(cmd) > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ > lib/python2.7/distutils/dist.py", > >>>>> >> >> line 972, in run_command > >>>>> >> >> cmd_obj.run() > >>>>> >> >> File "setup.py", line 142, in run > >>>>> >> >> super(cmd, self).run() > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ > proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", > >>>>> >> >> line 225, in run > >>>>> >> >> with self.project_on_sys_path(): > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ > lib/python2.7/contextlib.py", > >>>>> >> >> line 17, in __enter__ > >>>>> >> >> return self.gen.next() > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ > proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", > >>>>> >> >> line 164, in project_on_sys_path > >>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version)) > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ > proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", > >>>>> >> >> line 984, in require > >>>>> >> >> needed = self.resolve(parse_requirements(requirements)) > >>>>> >> >> File > >>>>> >> >> > >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ > proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", > >>>>> >> >> line 875, in resolve > >>>>> >> >> raise VersionConflict(dist, req).with_context(dependent_ > req) > >>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1 > >>>>> >> >> > >>>>> >> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/ > sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg), > >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), > >>>>> >> >> set(['apache-beam'])) > >>>>> >> >> Seems like grpcio did a release today which is breaking us: > >>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1 > >>>>> >> >> We have to either bump our protobuf dependency or reduce the > >>>>> >> >> upper > >>>>> >> >> bound of grpcio dependency to previous release (1.7.3). > >>>>> >> > > >>>>> >> > > >>>>> >> > > >>>>> >> > -- > >>>>> >> > This message was sent by Atlassian JIRA > >>>>> >> > (v6.4.14#64029) > >>>>> > > >>>>> > >
