[
https://issues.apache.org/jira/browse/AVRO-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris D'Urso updated AVRO-1486:
-------------------------------
Description:
Apparently unwanted status information is being printed out to standard output.
This interferes with the '\-' option when attempting to direct aver encoded
binary data to a pipe. A possible workaround is to use a named pipe instead,
but really the program should either be quiet with its status output at least
in the case where stdout is the file handle directed by the '\-' option.
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case,
the creation of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar
~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13
test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one 1
two 2
three 3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # now what should be
equivalent is not so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar
~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line"
is absent, and this is because its been written to stdout as well as the true
payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro
can be restored but not test2.txt.avro, plus test2 is bigger due to having
mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java
-jar ~/lib/avro-tools-1.7.6.jar totext - -
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java
-jar ~/lib/avro-tools-1.7.6.jar totext - -
Exception in thread "main" org.apache.avro.AvroRuntimeException:
java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
at
org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155
was:
Apparently unwanted status information is being printed out to standard output.
This interferes with the '-' option when attempting to direct aver encoded
binary data to a pipe. A possible workaround is to use a named pipe instead,
but really the program should either be quiet with its status output at least
in the case where stdout is the file handle directed by the '-' option.
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case,
the creation of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar
~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13
test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one 1
two 2
three 3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # now what should be
equivalent is not so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar
~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line"
is absent, and this is because its been written to stdout as well as the true
payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro
can be restored but not test2.txt.avro, plus test2 is bigger due to having
mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java
-jar ~/lib/avro-tools-1.7.6.jar totext - -
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java
-jar ~/lib/avro-tools-1.7.6.jar totext - -
Exception in thread "main" org.apache.avro.AvroRuntimeException:
java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
at
org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155
> avro-tools-1.7.6.jar fromtext to stdout fails because status also printed to
> stdout
> ------------------------------------------------------------------------------------
>
> Key: AVRO-1486
> URL: https://issues.apache.org/jira/browse/AVRO-1486
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.6
> Reporter: Chris D'Urso
>
> Apparently unwanted status information is being printed out to standard
> output. This interferes with the '\-' option when attempting to direct aver
> encoded binary data to a pipe. A possible workaround is to use a named pipe
> instead, but really the program should either be quiet with its status output
> at least in the case where stdout is the file handle directed by the '\-'
> option.
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning
> case, the creation of test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
> one 1
> two 2
> three 3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar
> ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - test.txt.avro
> Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
> WARNING: Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> Writing line = 5
> Writing line = 5
> Writing line = 7
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
> -rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13
> test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
> avro.codec
> snappy
> avro.schema
> "bytes"
> f@5:B
> one 1
> two 2
> three 3
> f@5:B
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
> 104 test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # now what should be
> equivalent is not so
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar
> ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - - > test2.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line"
> is absent, and this is because its been written to stdout as well as the true
> payload
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
> avro.codec
> snappy
> avro.schema
> "bytes"
> gD?Writing line = 5
> Writing line = 5
> Writing line = 7
> one 1
> two 2
> three 3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro
> can be restored but not test2.txt.avro, plus test2 is bigger due to having
> mixed stream data
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java
> -jar ~/lib/avro-tools-1.7.6.jar totext - -
> one 1
> two 2
> three 3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java
> -jar ~/lib/avro-tools-1.7.6.jar totext - -
> Exception in thread "main" org.apache.avro.AvroRuntimeException:
> java.io.IOException: Invalid sync!
> at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
> at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
> at org.apache.avro.tool.Main.run(Main.java:84)
> at org.apache.avro.tool.Main.main(Main.java:73)
> Caused by: java.io.IOException: Invalid sync!
> at
> org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
> at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
> ... 3 more
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
> 155
--
This message was sent by Atlassian JIRA
(v6.2#6252)
