GitHub user arunmahadevan opened a pull request:
https://github.com/apache/spark/pull/22251
[SPARK-25260][SQL] Fix namespace handling in SchemaConverters.toAvroType
## What changes were proposed in this pull request?
`toAvroType` converts spark data type to avro schema. It always appends the
record name to namespace so its impossible to have an Avro namespace
independent of the record name.
When invoked with a spark data type like,
```java
val sparkSchema = StructType(Seq(
StructField("name", StringType, nullable = false),
StructField("address", StructType(Seq(
StructField("city", StringType, nullable = false),
StructField("state", StringType, nullable = false))),
nullable = false)))
// map it to an avro schema with record name "employee" and top level
namespace "foo.bar",
val avroSchema = SchemaConverters.toAvroType(sparkSchema, false,
"employee", "foo.bar")
// result is
// avroSchema.getName = employee
// avroSchema.getNamespace = foo.bar.employee
// avroSchema.getFullname = foo.bar.employee.employee
```
The patch proposes to fix this so that the result is
```
avroSchema.getName = employee
avroSchema.getNamespace = foo.bar
avroSchema.getFullname = foo.bar.employee
```
## How was this patch tested?
New and existing unit tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arunmahadevan/spark avro-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22251
----
commit f47483951e12d563b7696940a2cfc2fdc3b27ab2
Author: Arun Mahadevan <arunm@...>
Date: 2018-08-28T08:00:17Z
[SPARK-25260][SQL] Fix namespace handling in SchemaConverters.toAvroType
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
