[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104570#comment-16104570 ] ASF subversion and git services commented on AVRO-2049: --- Commit 91d4cc03bbe84ca2aa0ed7f9094a20061b04a8a3 in avro's branch refs/heads/master from [~belugabehr] [ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=91d4cc0 ] AVRO-2049: Remove Superfluous Configuration From AvroSerializer > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch, AVRO-2049.3.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102841#comment-16102841 ] Gabor Szadovszky commented on AVRO-2049: +1 will commit tomorrow if no more comments [~nkollar], I did some research related to the usage of {{Serializer.open(OutputStream)}} and it seems they are not really reused. We might set the previous {{encoder}} to {{EncoderFactory.binaryEncoder(OutputStream, BinaryEncoder)}} to reuse, though. However, it would recommend having another JIRA for that. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch, AVRO-2049.3.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101827#comment-16101827 ] BELUGA BEHR commented on AVRO-2049: --- [~nkollar] Maybe so, we can reuse encoder. I'm not sure how often "open" is actually called though. New ticket? > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch, AVRO-2049.3.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101798#comment-16101798 ] BELUGA BEHR commented on AVRO-2049: --- [~gszadovszky] Good catch. Thanks. Perhaps we should create a new ticket to remove the magic number. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch, AVRO-2049.3.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101604#comment-16101604 ] Gabor Szadovszky commented on AVRO-2049: [~belugabehr], as far as I can see {{ResolvingGramarGenerator}} sets the _bufferSize_ (not the _blockSize_) which is used when a new _binaryEncoder_ is created. With your change the _bufferSize_ of the _binaryEncoder_ will be {{2048}} instead of {{32}} in {{ResolvingGrammarGenerator.getBinary}}. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101553#comment-16101553 ] Nandor Kollar commented on AVRO-2049: - Looks good to me, just one minor comment, in {{AvroSerialization}} I think we can reuse the {{encoder}} by passing it to the factory method instead of null: {{this.encoder = new EncoderFactory().binaryEncoder(out, encoder);}} > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092113#comment-16092113 ] Hadoop QA commented on AVRO-2049: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 46s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 7m 1s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2049 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877504/AVRO-2049.2.patch | | Optional Tests | buildtest javac | | uname | Linux 89a023614ebe 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/26/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[ https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092054#comment-16092054 ] Hadoop QA commented on AVRO-2049: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m 45s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m 35s{color} | {color:green} the patch passed {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a | | JIRA Issue | AVRO-2049 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12877504/AVRO-2049.2.patch | | Optional Tests | buildtest javac | | uname | Linux de48a9725531 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 GNU/Linux | | Build tool | build | | git revision | master / 793178a | | Default Java | 1.7.0_111 | | modules | C: lang/java U: lang/java | | Console output | https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/25/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated. > Remove Superfluous Configuration From AvroSerializer > > > Key: AVRO-2049 > URL: https://issues.apache.org/jira/browse/AVRO-2049 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch > > > In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the > Avro block size is configured with a hard-coded value and there is a request > to benchmark different buffer sizes. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > /** >* The block size for the Avro encoder. >* >* This number was copied from the AvroSerialization of > org.apache.avro.mapred in Avro 1.5.1. >* >* TODO(gwu): Do some benchmarking with different numbers here to see if it > is important. >*/ > private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512; > /** An factory for creating Avro datum encoders. */ > private static EncoderFactory mEncoderFactory > = new > EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES); > {code} > However, there is no need to benchmark, this setting is superfluous and is > ignored with the current implementation. > {code:title=org.apache.avro.hadoop.io.AvroSerializer} > @Override > public void open(OutputStream outputStream) throws IOException { > mOutputStream = outputStream; > mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder); > } > {code} > {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting. > This setting is only relevant for calls to > {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}} > which considers the configured "Block Size" for doing binary encoding of > blocked Array types as laid out in the > [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex]. > It can simply be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)