Re: Automate python formatting

[Michael A. Smith]

If all we want is the diff/exit code, then we should use pycodestyle
instead of black. I'll change course to work on that.

On Tue, May 28, 2019 at 5:24 AM Driesprong, Fokko 
wrote:

> Fully agree with you Ismaël.
>
>
> The issue is that it CAN write directly. For example here:
> https://github.com/Fokko/avro/pull/35 If you ask dependabot to merge it,
> it
> will be merged into master. And therefore it will ask for write access to
> the repository. Recently dependabot has been acquired
>  by Github, so I guess we can
> ask Github to sign a CLA :-) It is worth a shot of discussing this upstream
> in the ASF. Dependabot letting us know when there is a library with known
> CVE's will make the software much saver.
>
> Cheers, Fokko
>
> Op di 28 mei 2019 om 10:46 schreef Ismaël Mejía :
>
> > +1 for Black, great idea and in line we the changes we did with
> > spotless for the Java code base.
> >
> > About dependabot I think this is an interesting case to discuss in
> > upstream apache lists. So far ASF allows bots that do not touch the
> > code, for example for metadata, as we do with the autolabeler bot. If
> > I undertand correctly dependabot does not change the code, it opens
> > Pull Requests and it is up to a committer to decide or not if the code
> > is good. Or does it write directly?
> >
> > I suppose that even if technically is possible, there could be an
> > authorship issue to be discussed. Can bots sign an ICLA :D living in
> > the future mates!
> >
> > On Tue, May 28, 2019 at 10:01 AM Driesprong, Fokko  >
> > wrote:
> > >
> > > Thanks, Michael for working on this. I think having an auto formatter
> for
> > > Python is valuable since it will decrease the conflicts in the future.
> > For
> > > now, we need to do a big PR to get all the files in the correct format.
> > >
> > > We need to add to the CI: black --check --diff . This will throw a
> > non-zero
> > > exit code if there is something to format. So that the author of the PR
> > > needs to apply black to let the CI pass. We should make this part of
> the
> > > build.sh of the python3 project.
> > >
> > > I don't think that such a CI service exists, and I also think it is a
> bad
> > > idea. The author should format his code on forehand.
> > >
> > > I've also looked in setting up Dependabot for Avro, but this is against
> > the
> > > Apache rules because the Dependabot integration requires write
> > permissions
> > > on the repository which isn't allowed. Hope this helps.
> > >
> > > Cheers, Fokko
> > >
> > > Op di 28 mei 2019 om 03:39 schreef Michael A. Smith <
> > mich...@smith-li.com>:
> > >
> > > > I am working on making all the py and py3 code consistent with
> > > > https://github.com/python/black, but once done it’d be great if we
> > could
> > > > keep it consistent. I will look into adding hooks and stuff for
> > > > yetus/TravisCI, but is there a way to have an automation that can
> > > > periodically do all the formatting for us, and open pull requests
> with
> > any
> > > > changes required?
> > > >
> > > > I mean like dependabot, but instead of opening a pr to update
> > dependencies,
> > > > it opens a pr that does isort, black, docformatter, or whatever we
> > want.
> > > >
> > > > This way, we get consistent style without it being an “enforcement
> > > > priority” in prs by humans. If someone has a valuable contribution,
> we
> > > > don’t have to do a back-and-forth with them about style and
> formatting.
> > > >
> > > > If there’s interest I could look into implementing something with
> > existing
> > > > CI tools, or using github actions.
> > > >
> > > > What do y’all think?
> > > >
> >
>

[jira] [Commented] (AVRO-2400) Avro 1.9.0 can't resolve schemas that can be resolved in 1.8.2

[Jacob Tolar (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849928#comment-16849928
 ] 

Jacob Tolar commented on AVRO-2400:
---

{quote}Yes, good catch, lines 97 and 98 of SchemaCompatibility also need to be 
updated to use getName vs getFullName.
{quote}
I can certainly make this change, but I think we should at least consider the 
possibility that the pre-1.9 java resolver's behavior was 'wrong'. 
SchemaCompatibility thinks it's wrong; what about the other language 
implementations?. Was the behavior in 1.7.x and 1.8.x a regression from 
previous versions? (1.8.x is at least consistent with 1.5.3; I haven't checked 
any further back).

On the other hand, I think it's fair assume that Java is the most popularly 
used implementation, and that the resolving decoder is used significantly more 
than something like {{SchemaCompatibility}}. So you could make a strong 
argument for keeping 1.8.2's interpretation of the specification going forward. 
(This is essentially the approach we're considering here).

To be more succinct: we should be thoughtful/careful about changing the spec.

—

As an aside, it seems like the test cases in SchemaCompatibility or 
SchemaValidation should be runnable across all the language implementations. 
"Writer schema X should be readable by reader schema Y" shouldn't be 
Java-specific and maintaining a list of regression tests could help ensure all 
implementations interpret the spec in the same way.

> Avro 1.9.0 can't resolve schemas that can be resolved in 1.8.2
> --
>
> Key: AVRO-2400
> URL: https://issues.apache.org/jira/browse/AVRO-2400
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Reporter: Jacob Tolar
>Priority: Blocker
> Fix For: 1.10.0, 1.9.1
>
>
> The failure occurs in ResolvingGrammarGenerator when reader and writer schema 
> have an array of records with different full names (e.g. different 
> namespace). 
> {code:java}
> Exception in thread "main" java.lang.ClassCastException: 
> org.apache.avro.Resolver$ReaderUnion cannot be cast to 
> org.apache.avro.Resolver$Container{code}
> Avro 1.8.2 allowed this behavior but it now fails in 1.9.0. Looking at the 
> jiras and code, I don't believe this was intentional ( AVRO-2275,  
> [https://github.com/apache/avro/commit/39d959e1c6a1f339f03dab18289e47f27c10be7f]
>   ).
>  
> It looks like there were some attempts to keep compatibility ( 
> [https://github.com/apache/avro/blob/branch-1.9/lang/java/avro/src/main/java/org/apache/avro/Resolver.java]
>  , e.g. see the commented out check for w.getFullName() in resolve()) but 
> this case was missed.
>  
> See this simple example to reproduce. 
> [https://gist.github.com/jacobtolar/c88d43ab4e8767227891e5cdc188ffad]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Automate python formatting

[Driesprong, Fokko]

Fully agree with you Ismaël.


The issue is that it CAN write directly. For example here:
https://github.com/Fokko/avro/pull/35 If you ask dependabot to merge it, it
will be merged into master. And therefore it will ask for write access to
the repository. Recently dependabot has been acquired
 by Github, so I guess we can
ask Github to sign a CLA :-) It is worth a shot of discussing this upstream
in the ASF. Dependabot letting us know when there is a library with known
CVE's will make the software much saver.

Cheers, Fokko

Op di 28 mei 2019 om 10:46 schreef Ismaël Mejía :

> +1 for Black, great idea and in line we the changes we did with
> spotless for the Java code base.
>
> About dependabot I think this is an interesting case to discuss in
> upstream apache lists. So far ASF allows bots that do not touch the
> code, for example for metadata, as we do with the autolabeler bot. If
> I undertand correctly dependabot does not change the code, it opens
> Pull Requests and it is up to a committer to decide or not if the code
> is good. Or does it write directly?
>
> I suppose that even if technically is possible, there could be an
> authorship issue to be discussed. Can bots sign an ICLA :D living in
> the future mates!
>
> On Tue, May 28, 2019 at 10:01 AM Driesprong, Fokko 
> wrote:
> >
> > Thanks, Michael for working on this. I think having an auto formatter for
> > Python is valuable since it will decrease the conflicts in the future.
> For
> > now, we need to do a big PR to get all the files in the correct format.
> >
> > We need to add to the CI: black --check --diff . This will throw a
> non-zero
> > exit code if there is something to format. So that the author of the PR
> > needs to apply black to let the CI pass. We should make this part of the
> > build.sh of the python3 project.
> >
> > I don't think that such a CI service exists, and I also think it is a bad
> > idea. The author should format his code on forehand.
> >
> > I've also looked in setting up Dependabot for Avro, but this is against
> the
> > Apache rules because the Dependabot integration requires write
> permissions
> > on the repository which isn't allowed. Hope this helps.
> >
> > Cheers, Fokko
> >
> > Op di 28 mei 2019 om 03:39 schreef Michael A. Smith <
> mich...@smith-li.com>:
> >
> > > I am working on making all the py and py3 code consistent with
> > > https://github.com/python/black, but once done it’d be great if we
> could
> > > keep it consistent. I will look into adding hooks and stuff for
> > > yetus/TravisCI, but is there a way to have an automation that can
> > > periodically do all the formatting for us, and open pull requests with
> any
> > > changes required?
> > >
> > > I mean like dependabot, but instead of opening a pr to update
> dependencies,
> > > it opens a pr that does isort, black, docformatter, or whatever we
> want.
> > >
> > > This way, we get consistent style without it being an “enforcement
> > > priority” in prs by humans. If someone has a valuable contribution, we
> > > don’t have to do a back-and-forth with them about style and formatting.
> > >
> > > If there’s interest I could look into implementing something with
> existing
> > > CI tools, or using github actions.
> > >
> > > What do y’all think?
> > >
>

[jira] [Commented] (AVRO-2408) Make Forrest work again in the Docker image for local development

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849486#comment-16849486
 ] 

Hudson commented on AVRO-2408:
--

SUCCESS: Integrated in Jenkins build AvroJava #668 (See 
[https://builds.apache.org/job/AvroJava/668/])
AVRO-2408: Create local dev environment with Apache Forrest (#530) (fokko: 
[https://github.com/apache/avro/commit/41759e6c92d1d2578db0e878ead822a7f1abef99])
* (edit) share/docker/Dockerfile
* (edit) build.sh
* (add) share/docker/DockerfileLocal


> Make Forrest work again in the Docker image for local development
> -
>
> Key: AVRO-2408
> URL: https://issues.apache.org/jira/browse/AVRO-2408
> Project: Apache Avro
>  Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Automate python formatting

[Ismaël Mejía]

+1 for Black, great idea and in line we the changes we did with
spotless for the Java code base.

About dependabot I think this is an interesting case to discuss in
upstream apache lists. So far ASF allows bots that do not touch the
code, for example for metadata, as we do with the autolabeler bot. If
I undertand correctly dependabot does not change the code, it opens
Pull Requests and it is up to a committer to decide or not if the code
is good. Or does it write directly?

I suppose that even if technically is possible, there could be an
authorship issue to be discussed. Can bots sign an ICLA :D living in
the future mates!

On Tue, May 28, 2019 at 10:01 AM Driesprong, Fokko  wrote:
>
> Thanks, Michael for working on this. I think having an auto formatter for
> Python is valuable since it will decrease the conflicts in the future. For
> now, we need to do a big PR to get all the files in the correct format.
>
> We need to add to the CI: black --check --diff . This will throw a non-zero
> exit code if there is something to format. So that the author of the PR
> needs to apply black to let the CI pass. We should make this part of the
> build.sh of the python3 project.
>
> I don't think that such a CI service exists, and I also think it is a bad
> idea. The author should format his code on forehand.
>
> I've also looked in setting up Dependabot for Avro, but this is against the
> Apache rules because the Dependabot integration requires write permissions
> on the repository which isn't allowed. Hope this helps.
>
> Cheers, Fokko
>
> Op di 28 mei 2019 om 03:39 schreef Michael A. Smith :
>
> > I am working on making all the py and py3 code consistent with
> > https://github.com/python/black, but once done it’d be great if we could
> > keep it consistent. I will look into adding hooks and stuff for
> > yetus/TravisCI, but is there a way to have an automation that can
> > periodically do all the formatting for us, and open pull requests with any
> > changes required?
> >
> > I mean like dependabot, but instead of opening a pr to update dependencies,
> > it opens a pr that does isort, black, docformatter, or whatever we want.
> >
> > This way, we get consistent style without it being an “enforcement
> > priority” in prs by humans. If someone has a valuable contribution, we
> > don’t have to do a back-and-forth with them about style and formatting.
> >
> > If there’s interest I could look into implementing something with existing
> > CI tools, or using github actions.
> >
> > What do y’all think?
> >

Re: Automate python formatting

[Driesprong, Fokko]

Thanks, Michael for working on this. I think having an auto formatter for
Python is valuable since it will decrease the conflicts in the future. For
now, we need to do a big PR to get all the files in the correct format.

We need to add to the CI: black --check --diff . This will throw a non-zero
exit code if there is something to format. So that the author of the PR
needs to apply black to let the CI pass. We should make this part of the
build.sh of the python3 project.

I don't think that such a CI service exists, and I also think it is a bad
idea. The author should format his code on forehand.

I've also looked in setting up Dependabot for Avro, but this is against the
Apache rules because the Dependabot integration requires write permissions
on the repository which isn't allowed. Hope this helps.

Cheers, Fokko

Op di 28 mei 2019 om 03:39 schreef Michael A. Smith :

> I am working on making all the py and py3 code consistent with
> https://github.com/python/black, but once done it’d be great if we could
> keep it consistent. I will look into adding hooks and stuff for
> yetus/TravisCI, but is there a way to have an automation that can
> periodically do all the formatting for us, and open pull requests with any
> changes required?
>
> I mean like dependabot, but instead of opening a pr to update dependencies,
> it opens a pr that does isort, black, docformatter, or whatever we want.
>
> This way, we get consistent style without it being an “enforcement
> priority” in prs by humans. If someone has a valuable contribution, we
> don’t have to do a back-and-forth with them about style and formatting.
>
> If there’s interest I could look into implementing something with existing
> CI tools, or using github actions.
>
> What do y’all think?
>

[jira] [Commented] (AVRO-2408) Make Forrest work again in the Docker image for local development

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849438#comment-16849438
 ] 

ASF subversion and git services commented on AVRO-2408:
---

Commit 41759e6c92d1d2578db0e878ead822a7f1abef99 in avro's branch 
refs/heads/master from Niels Basjes
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=41759e6 ]

AVRO-2408: Create local dev environment with Apache Forrest (#530)



> Make Forrest work again in the Docker image for local development
> -
>
> Key: AVRO-2408
> URL: https://issues.apache.org/jira/browse/AVRO-2408
> Project: Apache Avro
>  Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2408) Make Forrest work again in the Docker image for local development

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2408.

   Resolution: Fixed
Fix Version/s: 1.10.0

> Make Forrest work again in the Docker image for local development
> -
>
> Key: AVRO-2408
> URL: https://issues.apache.org/jira/browse/AVRO-2408
> Project: Apache Avro
>  Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)