Automate python formatting
I am working on making all the py and py3 code consistent with https://github.com/python/black, but once done it’d be great if we could keep it consistent. I will look into adding hooks and stuff for yetus/TravisCI, but is there a way to have an automation that can periodically do all the formatting for us, and open pull requests with any changes required? I mean like dependabot, but instead of opening a pr to update dependencies, it opens a pr that does isort, black, docformatter, or whatever we want. This way, we get consistent style without it being an “enforcement priority” in prs by humans. If someone has a valuable contribution, we don’t have to do a back-and-forth with them about style and formatting. If there’s interest I could look into implementing something with existing CI tools, or using github actions. What do y’all think?
[jira] [Commented] (AVRO-2400) Avro 1.9.0 can't resolve schemas that can be resolved in 1.8.2
[
https://issues.apache.org/jira/browse/AVRO-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849231#comment-16849231
]
Raymie Stata commented on AVRO-2400:
[~jtolar] made the following observation via email:
bq. Note that the resolver and SchemaCompatibility disagree on this. (The test
provided would be marked 'incompatible' by SchemaCompatibility...and would have
been in 1.8.x as well, even though the actual parser/resolver allowed it). If
the spec is indeed updated per @rstata's suggestion then
SchemaCompatibility.java should probably be updated to match as well.
Yes, good catch, lines 97 and 98 of SchemaCompatibility also need to be updated
to use getName vs getFullName. Note, however, that line 102 needs to use the
writer's _full,_ because the aliasing part of the spec is clear as to what to
do with qualified vs unqualified names.
> Avro 1.9.0 can't resolve schemas that can be resolved in 1.8.2
> --
>
> Key: AVRO-2400
> URL: https://issues.apache.org/jira/browse/AVRO-2400
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Jacob Tolar
>Priority: Blocker
> Fix For: 1.10.0, 1.9.1
>
>
> The failure occurs in ResolvingGrammarGenerator when reader and writer schema
> have an array of records with different full names (e.g. different
> namespace).
> {code:java}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.avro.Resolver$ReaderUnion cannot be cast to
> org.apache.avro.Resolver$Container{code}
> Avro 1.8.2 allowed this behavior but it now fails in 1.9.0. Looking at the
> jiras and code, I don't believe this was intentional ( AVRO-2275,
> [https://github.com/apache/avro/commit/39d959e1c6a1f339f03dab18289e47f27c10be7f]
> ).
>
> It looks like there were some attempts to keep compatibility (
> [https://github.com/apache/avro/blob/branch-1.9/lang/java/avro/src/main/java/org/apache/avro/Resolver.java]
> , e.g. see the commented out check for w.getFullName() in resolve()) but
> this case was missed.
>
> See this simple example to reproduce.
> [https://gist.github.com/jacobtolar/c88d43ab4e8767227891e5cdc188ffad]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2410) Autoformat Python
[ https://issues.apache.org/jira/browse/AVRO-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849127#comment-16849127 ] Michael A. Smith commented on AVRO-2410: Implemented in [PR #532|https://github.com/apache/avro/pull/532] > Autoformat Python > - > > Key: AVRO-2410 > URL: https://issues.apache.org/jira/browse/AVRO-2410 > Project: Apache Avro > Issue Type: Task >Reporter: Michael A. Smith >Priority: Major > > I want to enable the .editorconfig for python (which is current commented > out.) To do that, we need to settle on some kind of consistency in both > python3 and python lang bases. The formatting tool > [black|https://github.com/python/black] is somewhat of a de-facto standard > now, but doesn't support python-2-only syntax, so in order to get there, we > need to fix the lang/py codebase. > I intend to do the minimum to get formatting standardized – not to attempt to > fully modernize the python codebase, but to maintain the existing > functionality and make it so black doesn't error out parsing it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2410) Autoformat Python
Michael A. Smith created AVRO-2410: -- Summary: Autoformat Python Key: AVRO-2410 URL: https://issues.apache.org/jira/browse/AVRO-2410 Project: Apache Avro Issue Type: Task Reporter: Michael A. Smith I want to enable the .editorconfig for python (which is current commented out.) To do that, we need to settle on some kind of consistency in both python3 and python lang bases. The formatting tool [black|https://github.com/python/black] is somewhat of a de-facto standard now, but doesn't support python-2-only syntax, so in order to get there, we need to fix the lang/py codebase. I intend to do the minimum to get formatting standardized – not to attempt to fully modernize the python codebase, but to maintain the existing functionality and make it so black doesn't error out parsing it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2409) Add Github PR template
Fokko Driesprong created AVRO-2409: -- Summary: Add Github PR template Key: AVRO-2409 URL: https://issues.apache.org/jira/browse/AVRO-2409 Project: Apache Avro Issue Type: Improvement Components: misc Affects Versions: 1.9.0 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 1.10.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2328) Support distinguishing between LocalDateTime and Instant semantics in timestamps
[ https://issues.apache.org/jira/browse/AVRO-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2328: Status: Patch Available (was: Open) > Support distinguishing between LocalDateTime and Instant semantics in > timestamps > > > Key: AVRO-2328 > URL: https://issues.apache.org/jira/browse/AVRO-2328 > Project: Apache Avro > Issue Type: Task >Reporter: Zoltan Ivanfi >Assignee: Nandor Kollar >Priority: Major > > Different SQL engines of the Hadoop stack support different timestamp > semantics. The range of supported semantics is about to be extended even > further. While some of the new timestamp types can be added to SQL without > explicit support from the file formats, others require new physical types. > File format support would be beneficial even for timestamp semantics where it > is not strictly required, because it would enable correct interpretation > without an SQL schema or any other kind of manual configuration. > This JIRA is about supporting the LocalDateTime and Instant semantics. See > [this > document|https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit#] > for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2405) Make Travis green again
[ https://issues.apache.org/jira/browse/AVRO-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848929#comment-16848929 ] Hudson commented on AVRO-2405: -- SUCCESS: Integrated in Jenkins build AvroJava #667 (See [https://builds.apache.org/job/AvroJava/667/]) AVRO-2405 Make Travis green again (#527) (github: [https://github.com/apache/avro/commit/48e518f5e7ad0e1bd456537e6ae0e640d5a46ef2]) * (edit) lang/java/maven-plugin/src/test/resources/unit/schema/pom-jsr310.xml * (edit) share/docker/Dockerfile * (edit) lang/java/maven-plugin/src/test/resources/unit/protocol/pom-joda.xml * (edit) share/VERSION.txt * (edit) lang/java/maven-plugin/src/test/resources/unit/protocol/pom-jsr310.xml * (edit) lang/java/maven-plugin/src/test/resources/unit/schema/pom-joda.xml * (edit) .travis.yml * (edit) lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml * (edit) lang/java/maven-plugin/src/test/resources/unit/idl/pom-jsr310.xml > Make Travis green again > --- > > Key: AVRO-2405 > URL: https://issues.apache.org/jira/browse/AVRO-2405 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2405) Make Travis green again
[ https://issues.apache.org/jira/browse/AVRO-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848900#comment-16848900 ] ASF subversion and git services commented on AVRO-2405: --- Commit 48e518f5e7ad0e1bd456537e6ae0e640d5a46ef2 in avro's branch refs/heads/master from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=48e518f ] AVRO-2405 Make Travis green again (#527) * AVRO-2405 Make Travis green again * Disable Forest for JDK11 * Remove Apache Forrest from the Dockerfile * Remove Forrest references > Make Travis green again > --- > > Key: AVRO-2405 > URL: https://issues.apache.org/jira/browse/AVRO-2405 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2405) Make Travis green again
[ https://issues.apache.org/jira/browse/AVRO-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848901#comment-16848901 ] ASF subversion and git services commented on AVRO-2405: --- Commit 48e518f5e7ad0e1bd456537e6ae0e640d5a46ef2 in avro's branch refs/heads/master from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=48e518f ] AVRO-2405 Make Travis green again (#527) * AVRO-2405 Make Travis green again * Disable Forest for JDK11 * Remove Apache Forrest from the Dockerfile * Remove Forrest references > Make Travis green again > --- > > Key: AVRO-2405 > URL: https://issues.apache.org/jira/browse/AVRO-2405 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2405) Make Travis green again
[ https://issues.apache.org/jira/browse/AVRO-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AVRO-2405. Resolution: Fixed > Make Travis green again > --- > > Key: AVRO-2405 > URL: https://issues.apache.org/jira/browse/AVRO-2405 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2408) Make Forrest work again in the Docker image for local development
Niels Basjes created AVRO-2408: -- Summary: Make Forrest work again in the Docker image for local development Key: AVRO-2408 URL: https://issues.apache.org/jira/browse/AVRO-2408 Project: Apache Avro Issue Type: Bug Reporter: Niels Basjes Assignee: Niels Basjes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2407) Automate the version bumping of the test resources
Fokko Driesprong created AVRO-2407: -- Summary: Automate the version bumping of the test resources Key: AVRO-2407 URL: https://issues.apache.org/jira/browse/AVRO-2407 Project: Apache Avro Issue Type: Improvement Components: build Affects Versions: 1.9.0 Reporter: Fokko Driesprong Fix For: 1.10.0 When performing a version bump using Maven, the pom's in the test directories aren't updated. For example lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml. This is a source of errors and confusion, therefore we want to automatically update these versions as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2406) Remove manual updating of the VERSION.txt
Fokko Driesprong created AVRO-2406: -- Summary: Remove manual updating of the VERSION.txt Key: AVRO-2406 URL: https://issues.apache.org/jira/browse/AVRO-2406 Project: Apache Avro Issue Type: Improvement Components: build Affects Versions: 1.9.0 Reporter: Fokko Driesprong Fix For: 1.10.0 Right now we're still manually updating the VERSION.txt: https://cwiki.apache.org/confluence/display/AVRO/How+To+Release The process is cumbersome and error-prone. It would be better to automate the detection of this version. For example, generate this by extracting the version from the pom.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2405) Make Travis green again
[ https://issues.apache.org/jira/browse/AVRO-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848692#comment-16848692 ] Niels Basjes commented on AVRO-2405: I think I may have found the cause: In some places it still says 1.9.0-SNAPSHOT [https://github.com/nielsbasjes/avro/commit/73acffe5d20f32559665d53f3e2e060b677818e9] > Make Travis green again > --- > > Key: AVRO-2405 > URL: https://issues.apache.org/jira/browse/AVRO-2405 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
