[jira] [Commented] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092124#comment-16092124 ] Hadoop QA commented on AVRO-2051: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 6s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} buildtest {color} | {color:red} 0m 23s{color} | {color:red} java in the patch failed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 2m 34s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2051 | | GITHUB PR | https://github.com/apache/avro/pull/236 | | Optional Tests | buildtest javac | | uname | Linux 45aa718b6e74 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | buildtest | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/29/artifact/patchprocess/test--lang_java.txt | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/29/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092113#comment-16092113 ] Hadoop QA commented on AVRO-2049: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 46s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 1s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2049 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877504/AVRO-2049.2.patch | | Optional Tests | buildtest javac | | uname | Linux 89a023614ebe 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/26/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2050) Clear Array To Allow GC
[ https://issues.apache.org/jira/browse/AVRO-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092131#comment-16092131 ] Hadoop QA commented on AVRO-2050: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 52s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2050 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877874/AVRO-2050.3.patch | | Optional Tests | buildtest javac | | uname | Linux ec17bb8bd028 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/27/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Clear Array To Allow GC > --- > > Key: AVRO-2050 > URL: https://issues.apache.org/jira/browse/AVRO-2050 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: AVRO-2050.1.patch, AVRO-2050.2.patch, AVRO-2050.3.patch > > > Java's {{ArrayList}} implementation clears all Objects from the internal > buffer when the {{clear()}} method is called. This allows the Objects to be > free for GC. We should do the same in Avro > {{org.apache.avro.generic.GenericData}} > [ArrayList > Source|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/ArrayList.java#ArrayList.clear%28%29] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092054#comment-16092054 ] Hadoop QA commented on AVRO-2049: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m 45s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 35s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2049 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877504/AVRO-2049.2.patch | | Optional Tests | buildtest javac | | uname | Linux de48a9725531 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/25/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2055) Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile
[ https://issues.apache.org/jira/browse/AVRO-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095813#comment-16095813 ] Hadoop QA commented on AVRO-2055: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 35s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 6m 50s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2055 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877868/AVRO-2055.1.patch | | Optional Tests | buildtest javac | | uname | Linux 7f5310dc7da6 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/33/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile > -- > > Key: AVRO-2055 > URL: https://issues.apache.org/jira/browse/AVRO-2055 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2055.1.patch > > > Remove magic string _io.file.buffer.size_ and _DEFAULT_BUFFER_SIZE_BYTES_ and > instead rely on the Hadoop libraries to provide this information. Will help > to keep Avro in sync with changes in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2048) Avro Binary Decoding - Gracefully Handle Long Strings
[ https://issues.apache.org/jira/browse/AVRO-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095803#comment-16095803 ] Hadoop QA commented on AVRO-2048: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} buildtest {color} | {color:red} 1m 36s{color} | {color:red} java in the patch failed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 1m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2048 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877398/AVRO-2048.1.patch | | Optional Tests | buildtest javac | | uname | Linux 8139c24a523e 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | buildtest | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/32/artifact/patchprocess/test--lang_java.txt | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/32/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Avro Binary Decoding - Gracefully Handle Long Strings > - > > Key: AVRO-2048 > URL: https://issues.apache.org/jira/browse/AVRO-2048 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: AVRO-2048.1.patch > > > According to the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_primitive]: > bq. a string is encoded as a *long* followed by that many bytes of UTF-8 > encoded character data. > However, that is currently not being adhered to: > {code:title=org.apache.avro.io.BinaryDecoder} > @Override > public Utf8 readString(Utf8 old) throws IOException { > int length = readInt(); > Utf8 result = (old != null ? old : new Utf8()); > result.setByteLength(length); > if (0 != length) { > doReadBytes(result.getBytes(), 0, length); > } > return result; > } > {code} > The first thing the code does here is to load an *int* value, not a *long*. > Because of the variable length nature of the size, this will mostly work. > However, there may be edge-cases where the serializer is putting in large > length values erroneously or nefariously. Let us gracefully detect such > scenarios and more closely adhere to the spec. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2054) Use StringBuilder instead of StringBuffer
[ https://issues.apache.org/jira/browse/AVRO-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095801#comment-16095801 ] Hadoop QA commented on AVRO-2054: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 35s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 6m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2054 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877859/AVRO-2054.1.patch | | Optional Tests | buildtest javac | | uname | Linux 009ba9974657 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/31/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Use StringBuilder instead of StringBuffer > - > > Key: AVRO-2054 > URL: https://issues.apache.org/jira/browse/AVRO-2054 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2054.1.patch > > > Use the un-synchronized StringBuilder instead of StringBuffer. Use _char_ > values instead of Strings. > {code:title=org.apache.trevni.MetaData} > @Override public String toString() { > StringBuffer buffer = new StringBuffer(); > buffer.append("{ "); > for (Map.Entrye : entrySet()) { > buffer.append(e.getKey()); > buffer.append("="); > try { > buffer.append(new String(e.getValue(), "ISO-8859-1")); > } catch (java.io.UnsupportedEncodingException error) { > throw new TrevniRuntimeException(error); > } > buffer.append(" "); > } > buffer.append("}"); > return buffer.toString(); > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093439#comment-16093439 ] Hadoop QA commented on AVRO-1485: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 1s{color} | {color:red} The patch has 1 ill-formed XML file(s). {color} | | {color:red}-1{color} | {color:red} buildtest {color} | {color:red} 15m 28s{color} | {color:red} root in the patch failed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 56s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-1485 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12874817/AVRO-1485_1.patch | | Optional Tests | buildtest xml | | uname | Linux 103d1460dcf7 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/30/artifact/patchprocess/whitespace-tabs.txt | | xml | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/30/artifact/patchprocess/xml.txt | | buildtest | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/30/artifact/patchprocess/test--root.txt | | modules | C: . U: . | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/30/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1957) TimeConversions do not implement getRecommendedSchema()
[ https://issues.apache.org/jira/browse/AVRO-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982264#comment-15982264 ] Hadoop QA commented on AVRO-1957: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 118 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 3s{color} | {color:red} The patch 16 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 41s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/avro:793178a | | JIRA Issue | AVRO-1957 | | GITHUB PR | https://github.com/apache/avro/pull/154 | | Optional Tests | buildtest javac | | uname | Linux 50a6b138572d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/4/artifact/patchprocess/whitespace-eol.txt | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/4/artifact/patchprocess/whitespace-tabs.txt | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/4/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > TimeConversions do not implement getRecommendedSchema() > --- > > Key: AVRO-1957 > URL: https://issues.apache.org/jira/browse/AVRO-1957 > Project: Avro > Issue Type: Bug >Affects Versions: 1.8.1 >Reporter: Sean Timm >Assignee: Sean Timm > Fix For: 1.8.2 > > > org.apache.avro.data.TimeConversions.TimestampConversion and other date and > time conversions do not implement getRecommendedSchema(). When trying to > dynamically generate an Avro schema from a pojo that contains a DateTime > object using ReflectData, I get an unsupported operation exception. > I think the implementation should be as simple as > {code} > @Override > public Schema getRecommendedSchema() { > return > LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG)); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-2016) Add support for adding include dirs for avdl -> avpr generation
[ https://issues.apache.org/jira/browse/AVRO-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982311#comment-15982311 ] Hadoop QA commented on AVRO-2016: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 96 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 3s{color} | {color:red} The patch 16 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 36s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/avro:793178a | | JIRA Issue | AVRO-2016 | | GITHUB PR | https://github.com/apache/avro/pull/206 | | Optional Tests | buildtest javac | | uname | Linux ca86621cc4a6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/5/artifact/patchprocess/whitespace-eol.txt | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/5/artifact/patchprocess/whitespace-tabs.txt | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/5/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Add support for adding include dirs for avdl -> avpr generation > --- > > Key: AVRO-2016 > URL: https://issues.apache.org/jira/browse/AVRO-2016 > Project: Avro > Issue Type: Improvement > Components: java >Reporter: Pepijn Noltes >Assignee: Pepijn Noltes >Priority: Minor > > I am using Apache Avro with a project setup where avdl files are imported by > project dependencies. > In this setup, the project directory containing the avdl files and the > directories containing the imported avdl files are separated. > During build I can bring these together, but during development (using > avroclipse) they are still separated. I would like to specify which > additional include dirs the 'avro-tools idl' should look for when resolving a > 'import idl' statement. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1976) Add Input/OutputFormat to read/write encoded objects
[ https://issues.apache.org/jira/browse/AVRO-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987973#comment-15987973 ] Hadoop QA commented on AVRO-1976: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 52s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 22s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-1976 | | GITHUB PR | https://github.com/apache/avro/pull/182 | | Optional Tests | buildtest javac | | uname | Linux 05582b3c96e1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/11/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Add Input/OutputFormat to read/write encoded objects > > > Key: AVRO-1976 > URL: https://issues.apache.org/jira/browse/AVRO-1976 > Project: Avro > Issue Type: Improvement > Components: java > Environment: hadoop >Reporter: Marius Posta >Priority: Minor > Labels: newbie > Original Estimate: 1h > Remaining Estimate: 1h > > In certain cases, performance of some Avro map-reduce jobs can be > considerably improved by de-coupling Avro encoding from actual Avro container > file IO. > In my case, a complex schema (100+ record fields) and large HDFS blocks > resulted in Spark jobs where a lot of workers were idling while a couple of > them were busy decoding their input splits.Furthermore, the objects then > needed to be re-encoded in order to be shuffled about, which crippled > performance further. > I propose the addition of an InputFormat which reads a container file input > split as key-value pairs in which the key is the file header and the value is > the decompressed file data block. Also, an OutputFormat which follows the > same logic for writing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[ https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988082#comment-15988082 ] Hadoop QA commented on AVRO-2003: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 0s{color} | {color:red} java in the patch failed. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 39s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 2s{color} | {color:red} java in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 1s{color} | {color:red} java in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 1s{color} | {color:red} java in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 21s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 2s{color} | {color:red} java in the patch failed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2003 | | GITHUB PR | https://github.com/apache/avro/pull/201 | | Optional Tests | buildtest javac compile javadoc | | uname | Linux 5ed3aa5d1014 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | compile | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt | | javadoc | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/javadoc--lang_java.txt | | compile | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt | | javac | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/whitespace-eol.txt | | javadoc | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/javadoc--lang_java.txt | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Report specific location of schema incompatibilities > > > Key: AVRO-2003 > URL: https://issues.apache.org/jira/browse/AVRO-2003 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.1 > Environment: Any java env >Reporter: Elliot West >Assignee: Elliot West >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-2003.patch > > > h2. Overview > Building on the work to improve schema incompatibility reporting in > AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could > also report the location in the schema where any incompatibility was > encountered. > It is recommended that the location reported is both easily readable by > humans and machines. In the first case this would assist schema developers to > pin-point issues in there schema documents, and in the latter case it > provides a useful mechanism to schema tooling, such as IDEs and editors, to > easily select the pertinent nodes in the Schema document tree. > h2. Implementation specifics > To meet this requirements it is suggested that the location is encoded using > the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is > both easily parsed by users, but is also supported by a number of libraries > for a range of common programming languages and platforms. > h2. Examples > Given the following example schema, consider some incompatibility scenarios. > For each case an expected JSON Pointer description of the incompatibility > location is described: > {code} > { >
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[ https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988093#comment-15988093 ] Hadoop QA commented on AVRO-2003: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 40s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2003 | | GITHUB PR | https://github.com/apache/avro/pull/201 | | Optional Tests | buildtest javac | | uname | Linux 8a7da17fc74b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | whitespace | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/15/artifact/patchprocess/whitespace-eol.txt | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/15/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Report specific location of schema incompatibilities > > > Key: AVRO-2003 > URL: https://issues.apache.org/jira/browse/AVRO-2003 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.1 > Environment: Any java env >Reporter: Elliot West >Assignee: Elliot West >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-2003.patch > > > h2. Overview > Building on the work to improve schema incompatibility reporting in > AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could > also report the location in the schema where any incompatibility was > encountered. > It is recommended that the location reported is both easily readable by > humans and machines. In the first case this would assist schema developers to > pin-point issues in there schema documents, and in the latter case it > provides a useful mechanism to schema tooling, such as IDEs and editors, to > easily select the pertinent nodes in the Schema document tree. > h2. Implementation specifics > To meet this requirements it is suggested that the location is encoded using > the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is > both easily parsed by users, but is also supported by a number of libraries > for a range of common programming languages and platforms. > h2. Examples > Given the following example schema, consider some incompatibility scenarios. > For each case an expected JSON Pointer description of the incompatibility > location is described: > {code} > { > "type": "record", > "name": "myRecord", > "fields" : [ > {"name": "pField", "type": "long"}, > {"name": "uField", "type": > ["null", "int", "string"] > }, > {"name": "eField", "type": > { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS", > "DIAMONDS", "CLUBS"] } > }, > {"name": "aField", "type": > {"type": "array", "items": "string"} > }, > {"name": "mField", "type": > {"type": "map", "values": "long"} > }, > {"name": "fField", "type": > {"type": "fixed", "size": 16, "name": "md5"} > } > ] > } > {code} > Possible incompatibility scenarions and the location that would be reported > back to the user/tool: > * Root type incompatibility; report location: {{/}} > * Record name mismatch; report location: {{/name}} > * {{pField}} type incompatibility; report location: {{/fields/0/type}} > * {{uField}} field type incompatibility; report location: {{/fields/1/type}} > * {{uField}} missing union branch {{string}}; report location: > {{/fields/1/type/2}} > * {{eField}} field type incompatibility; report location: {{/fields/2/type}} > * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}} > * {{eField}} enum name mismatch; report location:
[jira] [Commented] (AVRO-1047) Generated Java classes for specific records contain unchecked casts
[ https://issues.apache.org/jira/browse/AVRO-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186773#comment-16186773 ] Hadoop QA commented on AVRO-1047: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m 41s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/avro:9330bcb | | JIRA Issue | AVRO-1047 | | GITHUB PR | https://github.com/apache/avro/pull/250 | | Optional Tests | | | uname | Linux 77ac15314e50 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | branch-1.8 / 7ec35ea | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/35/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Generated Java classes for specific records contain unchecked casts > --- > > Key: AVRO-1047 > URL: https://issues.apache.org/jira/browse/AVRO-1047 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.6.3 >Reporter: Garrett Wu > Attachments: AVRO-1047.patch, suppress-warnings.tar.gz > > > The generated Java classes for specific records cause compiler warnings using > Oracle/Sun Java 1.6, since it doesn't support @SuppressWarnings("all"). > Instead could we change it to @SuppressWarnings("unchecked")? Only > "unchecked" and "deprecation" are mentioned Java Language Specification -- > the rest are specific to compiler vendors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)