On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <[email protected]> wrote:
> +1 for automating the process of checking for possible version bumps. > > Also, what do you think about pinning dependencies to exact versions > (instead of ranges) after cutting a release branch ? This should improve > the stability of released SDKs (but not a prefect solution since transitive > dependencies can still change). > This is a reasonable suggestion. The issue with that is, by being less flexible we will prevent users from using latest versions of dependencies. On the other hand it will prevent breaking of already released versions. > > Thanks, > Cham > > On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <[email protected]> wrote: > >> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <[email protected]> >> wrote: >> >>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <[email protected]> wrote: >>> > >>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <[email protected]> >>> > wrote: >>> >> >>> >> I am also in favor of pinning as an immediate fix, bumping the bound >>> >> otherwise. >>> >> >>> >> Regarding putting an upper bound to avoid being broken, the last two >>> >> breaks have been due to just having an (unneeded) upper bound (which >>> >> held us back to broken/incompatible releases in relationship to other >>> >> dependencies). We should try to trust semantic versioning when >>> >> possible, and when not we must regularly audit. >>> > >>> > +1 to this, especially the auditing part. We also had breaks because we >>> > trusted semantic versioning. So far our semi-official policy was to >>> trust a >>> > package until they prove it otherwise. I will argue that grpc here is >>> making >>> > a breaking change in a minor version increment by changing the way >>> they are >>> > depending on a major package. >>> >>> A minor version bump should be allowed to require a minor version bump >>> in its dependencies. >>> >>> > We have done a good job of auditing and updating those pinned (or upper >>> > bounded) dependencies, and probably we are behind in some of those. >>> > >>> > I wonder if we can automate some of this? If we can get a report, that >>> > audits our dependencies, warns us about new releases and potential >>> conflicts >>> > it would be much easier to keep things up to date. >>> >>> Big +1, it should be easy to set up a nightly that relaxes some of the >>> requirements and sees what (if anything) breaks. Not breaking is >>> likely a signal that we should relax ours. >>> >> >> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. I >> think it would be awesome if we can tackle this as part of a better >> infrastructure for testing work. >> >> >>> >>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA) >>> >> <[email protected]> wrote: >>> >> > >>> >> > [ >>> >> > https://issues.apache.org/jira/browse/BEAM-3357?page= >>> com.atlassian.jira.plugin.system.issuetabpanels:comment- >>> tabpanel&focusedCommentId=16293276#comment-16293276 >>> >> > ] >>> >> > >>> >> > Chamikara Jayalath commented on BEAM-3357: >>> >> > ------------------------------------------ >>> >> > >>> >> > You mean we could bump up the upper bound ? I think we should keep >>> some >>> >> > upper bound in case we get badly broken by a future protobuf >>> release. >>> >> > >>> >> >> Python SDK head fails to run tests due to >>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0') >>> >> >> >>> >> >> ------------------------------------------------------------ >>> -------------------------- >>> >> >> >>> >> >> Key: BEAM-3357 >>> >> >> URL: https://issues.apache.org/ >>> jira/browse/BEAM-3357 >>> >> >> Project: Beam >>> >> >> Issue Type: Bug >>> >> >> Components: sdk-py-core >>> >> >> Reporter: Chamikara Jayalath >>> >> >> Priority: Critical >>> >> >> >>> >> >> Error is: >>> >> >> running build_ext >>> >> >> Traceback (most recent call last): >>> >> >> File "setup.py", line 202, in <module> >>> >> >> 'test': generate_protos_first(test), >>> >> >> File >>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ >>> proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py", >>> >> >> line 129, in setup >>> >> >> return distutils.core.setup(**attrs) >>> >> >> File >>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ >>> lib/python2.7/distutils/core.py", >>> >> >> line 151, in setup >>> >> >> dist.run_commands() >>> >> >> File >>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ >>> lib/python2.7/distutils/dist.py", >>> >> >> line 953, in run_commands >>> >> >> self.run_command(cmd) >>> >> >> File >>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ >>> lib/python2.7/distutils/dist.py", >>> >> >> line 972, in run_command >>> >> >> cmd_obj.run() >>> >> >> File "setup.py", line 142, in run >>> >> >> super(cmd, self).run() >>> >> >> File >>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ >>> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", >>> >> >> line 225, in run >>> >> >> with self.project_on_sys_path(): >>> >> >> File >>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/ >>> lib/python2.7/contextlib.py", >>> >> >> line 17, in __enter__ >>> >> >> return self.gen.next() >>> >> >> File >>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ >>> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", >>> >> >> line 164, in project_on_sys_path >>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version)) >>> >> >> File >>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ >>> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", >>> >> >> line 984, in require >>> >> >> needed = self.resolve(parse_requirements(requirements)) >>> >> >> File >>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_ >>> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", >>> >> >> line 875, in resolve >>> >> >> raise VersionConflict(dist, req).with_context(dependent_req) >>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1 >>> >> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/ >>> sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg), >>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), >>> set(['apache-beam'])) >>> >> >> Seems like grpcio did a release today which is breaking us: >>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1 >>> >> >> We have to either bump our protobuf dependency or reduce the upper >>> >> >> bound of grpcio dependency to previous release (1.7.3). >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > This message was sent by Atlassian JIRA >>> >> > (v6.4.14#64029) >>> > >>> > >>> >>
