[
https://issues.apache.org/jira/browse/AVRO-3403?focusedWorklogId=747418&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-747418
]
ASF GitHub Bot logged work on AVRO-3403:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Mar/22 19:48
Start Date: 24/Mar/22 19:48
Worklog Time Spent: 10m
Work Description: martin-g commented on a change in pull request #1588:
URL: https://github.com/apache/avro/pull/1588#discussion_r834653670
##########
File path: lang/java/archetypes/avro-service-archetype/src/main/pom/pom.xml
##########
@@ -33,6 +33,8 @@
<name>Simple Avro Ordering Service</name>
<properties>
+ <maven.compiler.source>1.8</maven.compiler.source>
+ <maven.compiler.target>1.8</maven.compiler.target>
Review comment:
Should the values be references to the ones in the main/parent pom
(https://github.com/apache/avro/blob/fcc4e2dec64e8cbb011a388f5e612c2b09e4d3ef/pom.xml#L43-L44)
?
Either `${...}` or `@...@` (see
https://github.com/apache/wicket/blob/master/archetypes/quickstart/src/main/resources/archetype-resources/pom.xml#L46)
##########
File path: lang/java/idl/src/main/java/org/apache/avro/idl/Schemas.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.avro.idl;
+
+import org.apache.avro.Schema;
+import org.apache.avro.Schema.Field;
+
+import java.util.ArrayDeque;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+/**
+ * Avro Schema utilities, to traverse...
+ */
+public final class Schemas {
+
+ private Schemas() {
+ }
+
+ /**
+ * Depth first visit.
+ */
+ public static <T> T visit(final Schema start, final SchemaVisitor<T>
visitor) {
+ // Set of Visited Schemas
+ IdentityHashMap<Schema, Schema> visited = new IdentityHashMap<>();
+ // Stack that contains the Schemas to process and afterVisitNonTerminal
+ // functions.
+ // Deque<Either<Schema, Supplier<SchemaVisitorAction>>>
+ // Using Either<...> has a cost we want to avoid...
+ Deque<Object> dq = new ArrayDeque<>();
+ dq.push(start);
+ Object current;
+ while ((current = dq.poll()) != null) {
+ if (current instanceof Supplier) {
+ // We are executing a non-terminal post visit.
+ @SuppressWarnings("unchecked")
+ SchemaVisitorAction action = ((Supplier<SchemaVisitorAction>)
current).get();
+ switch (action) {
+ case CONTINUE:
+ break;
+ case SKIP_SUBTREE:
+ throw new UnsupportedOperationException();
Review comment:
add exception message ?!
##########
File path:
lang/java/tools/src/main/java/org/apache/avro/tool/IdlToSchemataTool.java
##########
@@ -38,24 +38,29 @@
public int run(InputStream in, PrintStream out, PrintStream err,
List<String> args) throws Exception {
if (args.isEmpty() || args.size() > 2 || isRequestingHelp(args)) {
err.println("Usage: idl2schemata [idl] [outdir]");
- err.println("");
+ err.println();
err.println("If an output directory is not specified, " + "outputs to
current directory.");
return -1;
}
boolean pretty = true;
- Idl parser = new Idl(new File(args.get(0)));
- File outputDirectory = getOutputDirectory(args);
-
- final Protocol protocol = parser.CompilationUnit();
- final List<String> warnings = parser.getWarningsAfterParsing();
+ IdlReader parser = new IdlReader();
Review comment:
Same question/suggestion as with the Maven Mojo above.
##########
File path: lang/java/idl/pom.xml
##########
@@ -0,0 +1,162 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";
+ xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <artifactId>avro-parent</artifactId>
+ <groupId>org.apache.avro</groupId>
+ <version>1.12.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>avro-idl</artifactId>
+
+ <name>Apache Avro IDL</name>
+ <packaging>bundle</packaging>
+ <url>https://avro.apache.org</url>
+ <description>Compilers for Avro IDL and Avro Specific Java API</description>
+
+ <properties>
+ <main.basedir>${project.parent.parent.basedir}</main.basedir>
+ <osgi.import>
+ !org.apache.avro.idl*,
+ org.apache.avro*;version="${project.version}",
+ org.apache.commons.text*,
+ *
+ </osgi.import>
+
<osgi.export>org.apache.avro.idl*;version="${project.version}"</osgi.export>
+ <antlr.version>4.9.3</antlr.version>
+ </properties>
+
+ <build>
+ <resources>
+ <resource>
+ <directory>src/main/resources</directory>
+ </resource>
+ </resources>
+ <testResources>
+ <testResource>
+ <directory>src/test/resources</directory>
+ </testResource>
+ <testResource>
+ <directory>src/test/idl</directory>
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <configuration>
+ <archive>
+ <manifestEntries>
+
<Automatic-Module-Name>org.apache.avro.idl</Automatic-Module-Name>
+ </manifestEntries>
+ </archive>
+ </configuration>
+ <executions>
+ <execution>
+ <id>prepare-test-jar</id>
+ <phase>generate-test-resources</phase>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ <configuration>
+ <classifier>test-resource</classifier>
+
<testClassesDirectory>src/test/idl/putOnClassPath</testClassesDirectory>
+ <finalName>putOnClassPath</finalName>
+
<outputDirectory>${project.build.testOutputDirectory}</outputDirectory>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.antlr</groupId>
+ <artifactId>antlr4-maven-plugin</artifactId>
+ <version>${antlr.version}</version>
+ <executions>
+ <execution>
+ <id>antlr</id>
+ <goals>
+ <goal>antlr4</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+
<sourceDirectory>${project.basedir}/../../../share/idl_grammar</sourceDirectory>
+
<libDirectory>${project.basedir}/../../../share/idl_grammar/imports</libDirectory>
+ <listener>true</listener>
+ <visitor>false</visitor>
+ </configuration>
+ </plugin>
+ </plugins>
+ <pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>org.eclipse.m2e</groupId>
Review comment:
This "plugin" should be defined in a profile so that it does not mess up
non-Eclipse users' experience.
See
https://github.com/apache/spark/commit/c7b0dd24176f9909031c9fbd151eb7f89fe56a59
##########
File path: lang/java/idl/src/test/idl/input/forward_ref.avdl
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+@namespace("org.foo")
+protocol Import {
+/* Name Value record */
+record ANameValue {
Review comment:
indentation ?
##########
File path:
lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/IDLProtocolMojo.java
##########
@@ -80,16 +80,16 @@ protected void doCompile(String filename, File
sourceDirectory, File outputDirec
}
}
- URLClassLoader projPathLoader = new
URLClassLoader(runtimeUrls.toArray(new URL[0]),
- Thread.currentThread().getContextClassLoader());
- try (Idl parser = new Idl(new File(sourceDirectory, filename),
projPathLoader)) {
-
- Protocol p = parser.CompilationUnit();
- for (String warning : parser.getWarningsAfterParsing()) {
+ final ClassLoader contextClassLoader =
Thread.currentThread().getContextClassLoader();
Review comment:
Would it be a good idea to keep the old (JavaCC) code in a separate Mojo
for a backup plan ?
I.e. if an application faces a bug in the new impl to be able to change the
Maven plugin goal and use the JavaCC impl ?
##########
File path: lang/java/idl/src/main/java/org/apache/avro/idl/IdlFile.java
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.avro.idl;
+
+import org.apache.avro.Protocol;
+import org.apache.avro.Schema;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * A parsed IdlFile. Provides access to the named schemas in the IDL file and
+ * the protocol containing the schemas.
+ */
+public class IdlFile {
+ private final Protocol protocol;
+ private final String namespace;
+ private final Map<String, Schema> namedSchemas;
+ private final List<String> warnings;
+
+ IdlFile(Protocol protocol, List<String> warnings) {
+ this(protocol.getNamespace(), protocol.getTypes(), protocol, warnings);
+ }
+
+ private IdlFile(String namespace, Iterable<Schema> schemas, Protocol
protocol, List<String> warnings) {
+ this.namespace = namespace;
+ this.namedSchemas = new LinkedHashMap<>();
+ for (Schema namedSchema : schemas) {
+ this.namedSchemas.put(namedSchema.getFullName(), namedSchema);
+ }
+ this.protocol = protocol;
+ this.warnings = Collections.unmodifiableList(new ArrayList<>(warnings));
+ }
+
+ /**
+ * The protocol defined by the IDL file.
+ */
+ public Protocol getProtocol() {
+ return protocol;
+ }
+
+ public List<String> getWarnings() {
+ return warnings;
+ }
+
+ public List<String> getWarnings(String importFile) {
+ return warnings.stream()
+ .map(warning -> importFile + " " +
Character.toLowerCase(warning.charAt(0)) + warning.substring(1))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * The default namespace to resolve schema names against.
+ */
+ public String getNamespace() {
+ return namespace;
+ }
+
+ /**
+ * The named schemas defined by the IDL file, mapped by their full name.
+ */
+ public Map<String, Schema> getNamedSchemas() {
+ return Collections.unmodifiableMap(namedSchemas);
+ }
+
+ /**
+ * Get a named schema defined by the IDL file, by name. The name can be a
simple
+ * name in the default namespace of the IDL file (e.g., the namespace of the
+ * protocol), or a full name.
+ *
+ * @param name the full name of the schema, or a simple name
+ * @return the schema, or {@code null} if it does not exist
+ */
+ public Schema getNamedSchema(String name) {
+ Schema result = namedSchemas.get(name);
+ if (result != null) {
+ return result;
+ }
+ if (namespace != null && !name.contains(".")) {
+ result = namedSchemas.get(namespace + "." + name);
+ }
+ return result;
+ }
+
+ // Visible for testing
+ String outputString() {
+ if (protocol != null) {
+ return protocol.toString();
+ }
+ if (namedSchemas.isEmpty()) {
+ return "[]";
+ } else {
+ StringBuilder buffer = new StringBuilder();
+ for (Schema schema : namedSchemas.values()) {
+ buffer.append(",").append(schema);
Review comment:
```suggestion
buffer.append(',').append(schema);
```
nit: it is a bit friendlier to memory allocation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 747418)
Time Spent: 0.5h (was: 20m)
> Migrate from JavaCC to ANTLR
> ----------------------------
>
> Key: AVRO-3403
> URL: https://issues.apache.org/jira/browse/AVRO-3403
> Project: Apache Avro
> Issue Type: Improvement
> Components: build, dependencies, java, tools
> Reporter: Oscar Westra van Holthe - Kind
> Assignee: Oscar Westra van Holthe - Kind
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Although JavaCC has been updated recently, there are quite some forks and
> development is quite erratic. The best maintained fork also has no presence
> in Maven Central yet. Worse (IMHO), it's Java only. This limits the use of
> the IDL format to Java and the Avro tools.
> As proposed on the mailling list last January, in the thread [Maintaining the
> IDL in the 21st
> century|https://lists.apache.org/thread/7c99hkkl59l78x5kyf7bd80cnyd1do3j],
> this improvement issue is to migrate the code to ANTLR.
> Related changes:
> # Place the ANTLR grammar in a subdirectory in the toplevel {{share}}
> directory, so it can be reused for other languages than Java (note: the
> Grammar may not contain any actions/code).
> # Ensure the IDL parsing API allows extending the IDL syntax to an {{.avsc}}
> equivalent.
> # Add a new {{java/idl}} module with the new parser. Make it a dependency of
> the {{java/compiler}} module and use it instead of the old parser.
> Bonus: this allows us to (also) create a parser that yields a single protocol
> (at first) or schema (future change).
> # Keep the old parser in the {{java/compiler}} module for backwards
> compatibility, mark it as deprecated and document its removal in a future
> version.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
