Agree that with the workaround for specifying specific versions, pinning exact versions becomes less useful. +1 for trusting semantic versioning and allowing dependencies to be updated till next major version whenever possible. If we run into issues we can ask users to pin specific versions when installing.
- Cham On Fri, Dec 15, 2017 at 3:09 PM Udi Meiri <[email protected]> wrote: > Yeah, I guess that's the difference between releasing source vs running > your own service. > > On Fri, Dec 15, 2017 at 2:58 PM Ahmet Altay <[email protected]> wrote: > >> I agree with Robert. Also there is usually a workaround to unbrake >> previous version by installing a specific version of an offending >> dependency. >> >> >> On Fri, Dec 15, 2017 at 2:53 PM, Robert Bradshaw <[email protected]> >> wrote: >> >>> This has the downside of pinning the dependencies for all downstream >>> projects, making it impossible for them to use different versions than >>> the ones we happened to choose. (Imagine the pain of two or more of >>> our dependencies pinned all their dependencies...) >>> >>> On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <[email protected]> wrote: >>> > +1 to pinning to exact versions, to be sure that our releases do not >>> break >>> > when newer versions of dependencies are released. >>> > >>> > On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <[email protected]> wrote: >>> >> >>> >> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath < >>> [email protected]> >>> >> wrote: >>> >>> >>> >>> +1 for automating the process of checking for possible version bumps. >>> >>> >>> >>> Also, what do you think about pinning dependencies to exact versions >>> >>> (instead of ranges) after cutting a release branch ? This should >>> improve the >>> >>> stability of released SDKs (but not a prefect solution since >>> transitive >>> >>> dependencies can still change). >>> >> >>> >> >>> >> This is a reasonable suggestion. The issue with that is, by being less >>> >> flexible we will prevent users from using latest versions of >>> dependencies. >>> >> On the other hand it will prevent breaking of already released >>> versions. >>> >> >>> >>> >>> >>> >>> >>> Thanks, >>> >>> Cham >>> >>> >>> >>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <[email protected]> >>> wrote: >>> >>>> >>> >>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw < >>> [email protected]> >>> >>>> wrote: >>> >>>>> >>> >>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <[email protected]> >>> wrote: >>> >>>>> > >>> >>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw >>> >>>>> > <[email protected]> >>> >>>>> > wrote: >>> >>>>> >> >>> >>>>> >> I am also in favor of pinning as an immediate fix, bumping the >>> bound >>> >>>>> >> otherwise. >>> >>>>> >> >>> >>>>> >> Regarding putting an upper bound to avoid being broken, the >>> last two >>> >>>>> >> breaks have been due to just having an (unneeded) upper bound >>> (which >>> >>>>> >> held us back to broken/incompatible releases in relationship to >>> >>>>> >> other >>> >>>>> >> dependencies). We should try to trust semantic versioning when >>> >>>>> >> possible, and when not we must regularly audit. >>> >>>>> > >>> >>>>> > +1 to this, especially the auditing part. We also had breaks >>> because >>> >>>>> > we >>> >>>>> > trusted semantic versioning. So far our semi-official policy was >>> to >>> >>>>> > trust a >>> >>>>> > package until they prove it otherwise. I will argue that grpc >>> here is >>> >>>>> > making >>> >>>>> > a breaking change in a minor version increment by changing the >>> way >>> >>>>> > they are >>> >>>>> > depending on a major package. >>> >>>>> >>> >>>>> A minor version bump should be allowed to require a minor version >>> bump >>> >>>>> in its dependencies. >>> >>>>> >>> >>>>> > We have done a good job of auditing and updating those pinned (or >>> >>>>> > upper >>> >>>>> > bounded) dependencies, and probably we are behind in some of >>> those. >>> >>>>> > >>> >>>>> > I wonder if we can automate some of this? If we can get a report, >>> >>>>> > that >>> >>>>> > audits our dependencies, warns us about new releases and >>> potential >>> >>>>> > conflicts >>> >>>>> > it would be much easier to keep things up to date. >>> >>>>> >>> >>>>> Big +1, it should be easy to set up a nightly that relaxes some of >>> the >>> >>>>> requirements and sees what (if anything) breaks. Not breaking is >>> >>>>> likely a signal that we should relax ours. >>> >>>> >>> >>>> >>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track >>> this. I >>> >>>> think it would be awesome if we can tackle this as part of a better >>> >>>> infrastructure for testing work. >>> >>>> >>> >>>>> >>> >>>>> >>> >>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA) >>> >>>>> >> <[email protected]> wrote: >>> >>>>> >> > >>> >>>>> >> > [ >>> >>>>> >> > >>> >>>>> >> > >>> https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276 >>> >>>>> >> > ] >>> >>>>> >> > >>> >>>>> >> > Chamikara Jayalath commented on BEAM-3357: >>> >>>>> >> > ------------------------------------------ >>> >>>>> >> > >>> >>>>> >> > You mean we could bump up the upper bound ? I think we should >>> keep >>> >>>>> >> > some >>> >>>>> >> > upper bound in case we get badly broken by a future protobuf >>> >>>>> >> > release. >>> >>>>> >> > >>> >>>>> >> >> Python SDK head fails to run tests due to >>> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0') >>> >>>>> >> >> >>> >>>>> >> >> >>> >>>>> >> >> >>> -------------------------------------------------------------------------------------- >>> >>>>> >> >> >>> >>>>> >> >> Key: BEAM-3357 >>> >>>>> >> >> URL: >>> >>>>> >> >> https://issues.apache.org/jira/browse/BEAM-3357 >>> >>>>> >> >> Project: Beam >>> >>>>> >> >> Issue Type: Bug >>> >>>>> >> >> Components: sdk-py-core >>> >>>>> >> >> Reporter: Chamikara Jayalath >>> >>>>> >> >> Priority: Critical >>> >>>>> >> >> >>> >>>>> >> >> Error is: >>> >>>>> >> >> running build_ext >>> >>>>> >> >> Traceback (most recent call last): >>> >>>>> >> >> File "setup.py", line 202, in <module> >>> >>>>> >> >> 'test': generate_protos_first(test), >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py", >>> >>>>> >> >> line 129, in setup >>> >>>>> >> >> return distutils.core.setup(**attrs) >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", >>> >>>>> >> >> line 151, in setup >>> >>>>> >> >> dist.run_commands() >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", >>> >>>>> >> >> line 953, in run_commands >>> >>>>> >> >> self.run_command(cmd) >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", >>> >>>>> >> >> line 972, in run_command >>> >>>>> >> >> cmd_obj.run() >>> >>>>> >> >> File "setup.py", line 142, in run >>> >>>>> >> >> super(cmd, self).run() >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", >>> >>>>> >> >> line 225, in run >>> >>>>> >> >> with self.project_on_sys_path(): >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", >>> >>>>> >> >> line 17, in __enter__ >>> >>>>> >> >> return self.gen.next() >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", >>> >>>>> >> >> line 164, in project_on_sys_path >>> >>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version)) >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", >>> >>>>> >> >> line 984, in require >>> >>>>> >> >> needed = self.resolve(parse_requirements(requirements)) >>> >>>>> >> >> File >>> >>>>> >> >> >>> >>>>> >> >> >>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", >>> >>>>> >> >> line 875, in resolve >>> >>>>> >> >> raise VersionConflict(dist, >>> req).with_context(dependent_req) >>> >>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf >>> 3.5.0.post1 >>> >>>>> >> >> >>> >>>>> >> >> >>> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg), >>> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), >>> >>>>> >> >> set(['apache-beam'])) >>> >>>>> >> >> Seems like grpcio did a release today which is breaking us: >>> >>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1 >>> >>>>> >> >> We have to either bump our protobuf dependency or reduce the >>> >>>>> >> >> upper >>> >>>>> >> >> bound of grpcio dependency to previous release (1.7.3). >>> >>>>> >> > >>> >>>>> >> > >>> >>>>> >> > >>> >>>>> >> > -- >>> >>>>> >> > This message was sent by Atlassian JIRA >>> >>>>> >> > (v6.4.14#64029) >>> >>>>> > >>> >>>>> > >>> >> >>
