[
https://issues.apache.org/jira/browse/SQOOP-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Senior updated SQOOP-3087:
--------------------------------
Description:
Created an Oracle table called {{SH_$.TEST}} and attempt to Sqoop import as
Avro ({{--as-avrodatafile}}) throws:
{code}
16/12/21 11:56:09 ERROR sqoop.Sqoop: Got exception running Sqoop:
org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
{code}
I believe this is because it is trying to use the value {{SH_$_TEST}} for the
{{"name"}} field in the Avro schema definition in the Avro data files. It seems
that this is not allowed as per the Avro specification on names:
https://avro.apache.org/docs/1.7.7/spec.html#Names
Command line is:
{code}
sqoop import --connect jdbc:oracle:thin:@localhost:1521/ORA11204 --username xxx
--password xxx --table SH_$.TEST --target-dir=/user/oracle/sh.db/test
--delete-target-dir -m2 --direct --fetch-size=5000 --as-avrodatafile
--outdir=/tmp
{code}
We know we can work around this by using the {{--query}} option for Sqoop, but
this means we are not able to benefit from the OraOop optimisations with direct
mode.
Can functionality (such as a schema conversion parameter) be added to Sqoop to
remove the {{$}} (and any other unsupported characters) from the {{name}} field
stored in the Avro schema in the Avro data file?
was:
Created an Oracle table called {{SH_$.TEST}} and attempt to Sqoop import as
Avro ({{--as-avrodatafile}}) throws:
{code}
16/12/21 11:56:09 ERROR sqoop.Sqoop: Got exception running Sqoop:
org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
{code}
I believe this is because it is trying to use the value {{SH_$_TEST}} for the
{{"name"}} field in the Avro schema definition in the Avro data files. It seems
that this is not allowed as per the Avro specification on names:
https://avro.apache.org/docs/1.7.7/spec.html#Names
Command line is:
{code}
sqoop import --connect jdbc:oracle:thin:@localhost:1521/ORA11204 --username xxx
--password xxx --table SH_$.TEST --target-dir=/user/oracle/sh.db/test
--delete-target-dir -m2 --direct --fetch-size=5000 --as-avrodatafile
--outdir=/tmp
{code}
We know we can work around this by using the {{--query}} option for Sqoop, but
this means we are not able to benefit from the OraOop optimisations with direct
mode.
Can functionality (such as a schema conversion parameter) be added to Sqoop to
remove the {{$}} from the {{name}} field stored in the Avro schema in the Avro
data file?
> Dollar ($) in Oracle schema name causes sqoop to avro to throw exception
> ------------------------------------------------------------------------
>
> Key: SQOOP-3087
> URL: https://issues.apache.org/jira/browse/SQOOP-3087
> Project: Sqoop
> Issue Type: Bug
> Affects Versions: 1.4.6
> Reporter: Steve Senior
>
> Created an Oracle table called {{SH_$.TEST}} and attempt to Sqoop import as
> Avro ({{--as-avrodatafile}}) throws:
> {code}
> 16/12/21 11:56:09 ERROR sqoop.Sqoop: Got exception running Sqoop:
> org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
> org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
> {code}
> I believe this is because it is trying to use the value {{SH_$_TEST}} for the
> {{"name"}} field in the Avro schema definition in the Avro data files. It
> seems that this is not allowed as per the Avro specification on names:
> https://avro.apache.org/docs/1.7.7/spec.html#Names
> Command line is:
> {code}
> sqoop import --connect jdbc:oracle:thin:@localhost:1521/ORA11204 --username
> xxx --password xxx --table SH_$.TEST --target-dir=/user/oracle/sh.db/test
> --delete-target-dir -m2 --direct --fetch-size=5000 --as-avrodatafile
> --outdir=/tmp
> {code}
> We know we can work around this by using the {{--query}} option for Sqoop,
> but this means we are not able to benefit from the OraOop optimisations with
> direct mode.
> Can functionality (such as a schema conversion parameter) be added to Sqoop
> to remove the {{$}} (and any other unsupported characters) from the {{name}}
> field stored in the Avro schema in the Avro data file?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
