Re: WELCOME to dev@avro.apache.org

[Thiruvalluvan MG]

 Hi Suraj,
Thanks for taking the initiative. I want to roll out a bunch of fixes for C++ 
issues pending for some time now. Please give me a a couple of days. I'll have 
them done.
Thank you,
Thiru
On Thursday, 22 March, 2018, 11:43:13 PM IST, Suraj Acharya 
 wrote:  
 
 Hi,
I have filed : https://issues.apache.org/jira/browse/AVRO-2163
If you believe any fix is needed for the release please link it.
If you want any change in please contribute.

Thanks
Suraj Acharya


On Thu, Mar 22, 2018 at 8:55 AM, Bridger Howell  wrote:

> Not a committer, but I'd like to chime in and say that I'd absolutely love
> to see another set of official releases soon.
>
> When looking through the blockers in JIRA, AVRO-1810 and AVRO-1891 stand
> out a bit - does anybody know if those issues should actually be considered
> "blockers"?
> Caveat here: I do have a personal interest in seeing AVRO-1891 fixed, and
> if we do decide that it is actually a blocker, I would definitely be
> willing to volunteer some time to help sort it out quickly.
>
> On Thu, Mar 22, 2018 at 8:55 AM, Edward Anderson 
> wrote:
>
> > Hi,
> >
> > There have been a lot of great improvements
> >  since
> May
> > 2017 when Avro 1.8.2 was released—241 commits from 55 different
> > contributors over nearly a year. We would love to see these included in a
> > new versioned release. Right now, people affected by this year's worth of
> > fixes need to work around the issues or do a local build of master, both
> of
> > which are inconvenient. The number of users affected will continue to
> grow
> > until the next release.
> >
> > Do you agree that it's time for a new release? If not now, when do you
> > think will be best? We wanted to check in before starting our own
> > workarounds.
> >
> > Thanks for all your work on this great project!
> >
> > Best,
> >
> > Edward Anderson
> > Software Engineer
> > Doximity, Inc.
> >
> > On Thu, Mar 22, 2018 at 10:54 AM,  wrote:
> >
> > > Hi! This is the ezmlm program. I'm managing the
> > > dev@avro.apache.org mailing list.
> > >
> > > I'm working for my owner, who can be reached
> > > at dev-ow...@avro.apache.org.
> > >
> > > Acknowledgment: I have added the address
> > >
> > >    eander...@doximity.com
> > >
> > > to the dev mailing list.
> > >
> > > Welcome to dev@avro.apache.org!
> > >
> > > Please save this message so that you know the address you are
> > > subscribed under, in case you later want to unsubscribe or change your
> > > subscription address.
> > >
> > >
> > > --- Administrative commands for the dev list ---
> > >
> > > I can handle administrative requests automatically. Please
> > > do not send them to the list address! Instead, send
> > > your message to the correct command address:
> > >
> > > To subscribe to the list, send a message to:
> > >    
> > >
> > > To remove your address from the list, send a message to:
> > >    
> > >
> > > Send mail to the following for info and FAQ for this list:
> > >    
> > >    
> > >
> > > Similar addresses exist for the digest list:
> > >    
> > >    
> > >
> > > To get messages 123 through 145 (a maximum of 100 per request), mail:
> > >    
> > >
> > > To get an index with subject and author for messages 123-456 , mail:
> > >    
> > >
> > > They are always returned as sets of 100, max 2000 per request,
> > > so you'll actually get 100-499.
> > >
> > > To receive all messages with the same subject as message 12345,
> > > send a short message to:
> > >    
> > >
> > > The messages should contain one line or word of text to avoid being
> > > treated as sp@m, but I will ignore their content.
> > > Only the ADDRESS you send to is important.
> > >
> > > You can start a subscription for an alternate address,
> > > for example "john@host.domain", just add a hyphen and your
> > > address (with '=' instead of '@') after the command word:
> > > 

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410747#comment-16410747
 ] 

ASF GitHub Bot commented on AVRO-1891:
--

wheelerlaw commented on issue #118: AVRO-1891: Fix specific nested logical types
URL: https://github.com/apache/avro/pull/118#issuecomment-375532605
 
 
   What happened to this? This seems pretty major, because without it, unions 
with built-in logical types are **broken**.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410223#comment-16410223
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176558658
 
 

 ##
 File path: lang/java/avro/src/test/java/org/apache/avro/file/TestCodecs.java
 ##
 @@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.avro.file;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Random;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+
+@RunWith(Parameterized.class)
+public class TestCodecs {
+
+  @Parameters(name = "Codec: {0}")
+  public static Collection data() {
+return Arrays.asList(new Object[][] {
+  { new NullCodec() },
+  { new DeflateCodec(3) },
+  { new BZip2Codec() },
+  { new SnappyCodec() },
+  { new XZCodec(3) },
+  { new ZstandardCodec(3) }
+});
+  }
+
+  private final Codec codec;
+  private final byte[] zeroes = new byte[1024*1024];
 
 Review comment:
   all zeroes tends to compress massively -- which can uncover bugs in buffer 
sizing when decompressing.
   
   pure random tends to be _larger_ when compressed than uncompressed and may 
find bugs in buffer sizing when compressing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410219#comment-16410219
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176558252
 
 

 ##
 File path: lang/java/avro/src/main/java/org/apache/avro/file/XZCodec.java
 ##
 @@ -57,62 +52,22 @@ public String getName() {
   }
 
   @Override
-  public ByteBuffer compress(ByteBuffer data) throws IOException {
-ByteArrayOutputStream baos = getOutputBuffer(data.remaining());
-OutputStream ios = new XZCompressorOutputStream(baos, compressionLevel);
-writeAndClose(data, ios);
-return ByteBuffer.wrap(baos.toByteArray());
+  protected OutputStream compressedStream(OutputStream output)
+  throws IOException {
+return new XZCompressorOutputStream(output, compressionLevel);
   }
 
   @Override
-  public ByteBuffer decompress(ByteBuffer data) throws IOException {
-ByteArrayOutputStream baos = getOutputBuffer(data.remaining());
-InputStream bytesIn = new ByteArrayInputStream(
-  data.array(),
-  data.arrayOffset() + data.position(),
-  data.remaining());
-InputStream ios = new XZCompressorInputStream(bytesIn);
-try {
-  IOUtils.copy(ios, baos);
-} finally {
-  ios.close();
-}
-return ByteBuffer.wrap(baos.toByteArray());
+  protected InputStream uncompressedStream(InputStream input)
+  throws IOException {
+return new XZCompressorInputStream(input);
   }
 
-  private void writeAndClose(ByteBuffer data, OutputStream to) throws 
IOException {
-byte[] input = data.array();
-int offset = data.arrayOffset() + data.position();
-int length = data.remaining();
-try {
-  to.write(input, offset, length);
-} finally {
-  to.close();
-}
-  }
-
-  // get and initialize the output buffer for use.
-  private ByteArrayOutputStream getOutputBuffer(int suggestedLength) {
-if (null == outputBuffer) {
-  outputBuffer = new ByteArrayOutputStream(suggestedLength);
-}
-outputBuffer.reset();
-return outputBuffer;
-  }
-
-  @Override
-  public int hashCode() {
-return compressionLevel;
-  }
+  @Override public int hashCode() { return getName().hashCode(); }
 
   @Override
-  public boolean equals(Object obj) {
-if (this == obj)
-  return true;
-if (getClass() != obj.getClass())
-  return false;
-XZCodec other = (XZCodec)obj;
 
 Review comment:
   The equals method for the XZ codec was wrong -- according to the 
specification of Codec, it should equal if it is mutually decompressible.  The 
compression level is used for the compressor but does not affect 
decompressibility.
   
   I made the implementation of hashCode and equals consistent across the 
codecs when appropriate.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410214#comment-16410214
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176557719
 
 

 ##
 File path: lang/java/avro/src/main/java/org/apache/avro/file/SnappyCodec.java
 ##
 @@ -34,50 +35,63 @@ protected Codec createInstance() {
 }
   }
 
-  private SnappyCodec() {}
+  SnappyCodec() {}
 
   @Override public String getName() { return DataFileConstants.SNAPPY_CODEC; }
 
   @Override
   public ByteBuffer compress(ByteBuffer in) throws IOException {
 
 Review comment:
   Tests were failing on only Snappy when I made some overly strict assumptions 
on the returned buffer. 
   
   In the process of debugging I fixed at least one bug (not setting 
ByteOrder.LITTLE_ENDIAN and letting the file format depend on the CPU of the 
writer).
   
   The code also was not properly accounting for arrayOffset in many cases.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410202#comment-16410202
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176556430
 
 

 ##
 File path: 
lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java
 ##
 @@ -351,16 +349,27 @@ ByteBuffer getAsByteBuffer() {
   return ByteBuffer.wrap(data, offset, blockSize);
 }
 
+void setBytes(ByteBuffer block) {
 
 Review comment:
   this now supports Codecs that return Direct ByteBuffers.  Earlier versions 
of the ZstandardCodec were using APIs that returned direct buffers, which 
exploded here. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410199#comment-16410199
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176555870
 
 

 ##
 File path: lang/java/avro/src/main/java/org/apache/avro/file/CodecFactory.java
 ##
 @@ -40,7 +42,9 @@
 public abstract class CodecFactory {
   /** Null codec, for no compression. */
   public static CodecFactory nullCodec() {
-return NullCodec.OPTION;
+// we can not reference NullCodec.OPTION because the static
 
 Review comment:
   The unit test uncovered the fact that accessing this field here results in 
'null' since we have a circular dependency in static initialization.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410198#comment-16410198
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176555679
 
 

 ##
 File path: lang/java/avro/src/main/java/org/apache/avro/file/BZip2Codec.java
 ##
 @@ -42,64 +40,20 @@ protected Codec createInstance() {
   public String getName() { return DataFileConstants.BZIP2_CODEC; }
 
   @Override
-  public ByteBuffer compress(ByteBuffer uncompressedData) throws IOException {
-
-ByteArrayOutputStream baos = getOutputBuffer(uncompressedData.remaining());
-BZip2CompressorOutputStream outputStream = new 
BZip2CompressorOutputStream(baos);
-
-try {
-  outputStream.write(uncompressedData.array(),
- uncompressedData.position(),
- uncompressedData.remaining());
-} finally {
-  outputStream.close();
-}
-
-ByteBuffer result = ByteBuffer.wrap(baos.toByteArray());
-return result;
+  protected OutputStream compressedStream(OutputStream output)
+  throws IOException {
+return new BZip2CompressorOutputStream(output);
   }
 
   @Override
-  public ByteBuffer decompress(ByteBuffer compressedData) throws IOException {
-ByteArrayInputStream bais = new 
ByteArrayInputStream(compressedData.array());
-BZip2CompressorInputStream inputStream = new 
BZip2CompressorInputStream(bais);
 
 Review comment:
   Most of the Codecs are internally based on InputStreams and OutputStreams.  
I refactored the commonalities out into two abstract classes (these would be 
better as 'mix-in' interfaces in Java 8+).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410195#comment-16410195
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on a change in pull request #303: AVRO-2162 Adds Zstandard 
compression to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#discussion_r176555302
 
 

 ##
 File path: lang/java/avro/pom.xml
 ##
 @@ -198,6 +198,11 @@
   joda-time
   true
 
+
+  com.github.luben
+  zstd-jni
+  true
 
 Review comment:
   I think that most of the compression dependencies should be .  Its 
extra baggage that is not useful in any case where we aren't writing out or 
reading files.  It would be more consistent with the other codecs to remove 
this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410193#comment-16410193
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey commented on issue #303: AVRO-2162 Adds Zstandard compression to the 
Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303#issuecomment-375438762
 
 
   While writing this and adding test coverage, I ended up making a few other 
clean-ups. 
   
   Do we need to add anything in the spec about this?  
   
   I also targetted the 1.8 branch because I assume that many people on older 
branches might be interested.  I'm still stuck on a 1.7.x branch in (hadoop) 
production myself -- though that should change in a few months.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410190#comment-16410190
 ] 

ASF GitHub Bot commented on AVRO-2162:
--

scottcarey opened a new pull request #303: AVRO-2162 Adds Zstandard compression 
to the Avro File Format (Java)
URL: https://github.com/apache/avro/pull/303
 
 
 Adds TestCodecs to cover all file compression Codecs.
 Consolidates common code in Codecs into OutputStreamCodec
 and OutputInputStreamCodec abstractions.
 Fixes DataFileStream so that Codecs can return DirectByteBuffers
 or Heap ByteBuffers with non-zero offset.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: WELCOME to dev@avro.apache.org

[Zoltan Farkas]

The following is also pretty low hanging fruit (one line change):

https://issues.apache.org/jira/browse/AVRO-2057


—Z

> On Mar 22, 2018, at 11:55 AM, Bridger Howell  wrote:
> 
> Not a committer, but I'd like to chime in and say that I'd absolutely love
> to see another set of official releases soon.
> 
> When looking through the blockers in JIRA, AVRO-1810 and AVRO-1891 stand
> out a bit - does anybody know if those issues should actually be considered
> "blockers"?
> Caveat here: I do have a personal interest in seeing AVRO-1891 fixed, and
> if we do decide that it is actually a blocker, I would definitely be
> willing to volunteer some time to help sort it out quickly.
> 
> On Thu, Mar 22, 2018 at 8:55 AM, Edward Anderson 
> wrote:
> 
>> Hi,
>> 
>> There have been a lot of great improvements
>>  since May
>> 2017 when Avro 1.8.2 was released—241 commits from 55 different
>> contributors over nearly a year. We would love to see these included in a
>> new versioned release. Right now, people affected by this year's worth of
>> fixes need to work around the issues or do a local build of master, both of
>> which are inconvenient. The number of users affected will continue to grow
>> until the next release.
>> 
>> Do you agree that it's time for a new release? If not now, when do you
>> think will be best? We wanted to check in before starting our own
>> workarounds.
>> 
>> Thanks for all your work on this great project!
>> 
>> Best,
>> 
>> Edward Anderson
>> Software Engineer
>> Doximity, Inc.
>> 
>> On Thu, Mar 22, 2018 at 10:54 AM,  wrote:
>> 
>>> Hi! This is the ezmlm program. I'm managing the
>>> dev@avro.apache.org mailing list.
>>> 
>>> I'm working for my owner, who can be reached
>>> at dev-ow...@avro.apache.org.
>>> 
>>> Acknowledgment: I have added the address
>>> 
>>>   eander...@doximity.com
>>> 
>>> to the dev mailing list.
>>> 
>>> Welcome to dev@avro.apache.org!
>>> 
>>> Please save this message so that you know the address you are
>>> subscribed under, in case you later want to unsubscribe or change your
>>> subscription address.
>>> 
>>> 
>>> --- Administrative commands for the dev list ---
>>> 
>>> I can handle administrative requests automatically. Please
>>> do not send them to the list address! Instead, send
>>> your message to the correct command address:
>>> 
>>> To subscribe to the list, send a message to:
>>>   
>>> 
>>> To remove your address from the list, send a message to:
>>>   
>>> 
>>> Send mail to the following for info and FAQ for this list:
>>>   
>>>   
>>> 
>>> Similar addresses exist for the digest list:
>>>   
>>>   
>>> 
>>> To get messages 123 through 145 (a maximum of 100 per request), mail:
>>>   
>>> 
>>> To get an index with subject and author for messages 123-456 , mail:
>>>   
>>> 
>>> They are always returned as sets of 100, max 2000 per request,
>>> so you'll actually get 100-499.
>>> 
>>> To receive all messages with the same subject as message 12345,
>>> send a short message to:
>>>   
>>> 
>>> The messages should contain one line or word of text to avoid being
>>> treated as sp@m, but I will ignore their content.
>>> Only the ADDRESS you send to is important.
>>> 
>>> You can start a subscription for an alternate address,
>>> for example "john@host.domain", just add a hyphen and your
>>> address (with '=' instead of '@') after the command word:
>>> 

Re: WELCOME to dev@avro.apache.org

[Suraj Acharya]

Hi,
I have filed : https://issues.apache.org/jira/browse/AVRO-2163
If you believe any fix is needed for the release please link it.
If you want any change in please contribute.

Thanks
Suraj Acharya


On Thu, Mar 22, 2018 at 8:55 AM, Bridger Howell  wrote:

> Not a committer, but I'd like to chime in and say that I'd absolutely love
> to see another set of official releases soon.
>
> When looking through the blockers in JIRA, AVRO-1810 and AVRO-1891 stand
> out a bit - does anybody know if those issues should actually be considered
> "blockers"?
> Caveat here: I do have a personal interest in seeing AVRO-1891 fixed, and
> if we do decide that it is actually a blocker, I would definitely be
> willing to volunteer some time to help sort it out quickly.
>
> On Thu, Mar 22, 2018 at 8:55 AM, Edward Anderson 
> wrote:
>
> > Hi,
> >
> > There have been a lot of great improvements
> >  since
> May
> > 2017 when Avro 1.8.2 was released—241 commits from 55 different
> > contributors over nearly a year. We would love to see these included in a
> > new versioned release. Right now, people affected by this year's worth of
> > fixes need to work around the issues or do a local build of master, both
> of
> > which are inconvenient. The number of users affected will continue to
> grow
> > until the next release.
> >
> > Do you agree that it's time for a new release? If not now, when do you
> > think will be best? We wanted to check in before starting our own
> > workarounds.
> >
> > Thanks for all your work on this great project!
> >
> > Best,
> >
> > Edward Anderson
> > Software Engineer
> > Doximity, Inc.
> >
> > On Thu, Mar 22, 2018 at 10:54 AM,  wrote:
> >
> > > Hi! This is the ezmlm program. I'm managing the
> > > dev@avro.apache.org mailing list.
> > >
> > > I'm working for my owner, who can be reached
> > > at dev-ow...@avro.apache.org.
> > >
> > > Acknowledgment: I have added the address
> > >
> > >eander...@doximity.com
> > >
> > > to the dev mailing list.
> > >
> > > Welcome to dev@avro.apache.org!
> > >
> > > Please save this message so that you know the address you are
> > > subscribed under, in case you later want to unsubscribe or change your
> > > subscription address.
> > >
> > >
> > > --- Administrative commands for the dev list ---
> > >
> > > I can handle administrative requests automatically. Please
> > > do not send them to the list address! Instead, send
> > > your message to the correct command address:
> > >
> > > To subscribe to the list, send a message to:
> > >
> > >
> > > To remove your address from the list, send a message to:
> > >
> > >
> > > Send mail to the following for info and FAQ for this list:
> > >
> > >
> > >
> > > Similar addresses exist for the digest list:
> > >
> > >
> > >
> > > To get messages 123 through 145 (a maximum of 100 per request), mail:
> > >
> > >
> > > To get an index with subject and author for messages 123-456 , mail:
> > >
> > >
> > > They are always returned as sets of 100, max 2000 per request,
> > > so you'll actually get 100-499.
> > >
> > > To receive all messages with the same subject as message 12345,
> > > send a short message to:
> > >
> > >
> > > The messages should contain one line or word of text to avoid being
> > > treated as sp@m, but I will ignore their content.
> > > Only the ADDRESS you send to is important.
> > >
> > > You can start a subscription for an alternate address,
> > > for example "john@host.domain", just add a hyphen and your
> > > address (with '=' instead of '@') after the command word:
> > > 

[jira] [Updated] (AVRO-2163) Release 1.8.3

[Suraj Acharya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suraj Acharya updated AVRO-2163:

Description: Please link to any issues that should be considered blockers 
for the 1.8.3 release.  (was: Please link to any issues that should be 
considered blockers for the 1.8.2 release.)

> Release 1.8.3
> -
>
> Key: AVRO-2163
> URL: https://issues.apache.org/jira/browse/AVRO-2163
> Project: Avro
>  Issue Type: Task
>  Components: community
>Affects Versions: 1.8.3
>Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Major
> Fix For: 1.8.3
>
>
> Please link to any issues that should be considered blockers for the 1.8.3 
> release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2163) Release 1.8.3

[Suraj Acharya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suraj Acharya updated AVRO-2163:

Fix Version/s: (was: 1.8.2)
   1.8.3

> Release 1.8.3
> -
>
> Key: AVRO-2163
> URL: https://issues.apache.org/jira/browse/AVRO-2163
> Project: Avro
>  Issue Type: Task
>  Components: community
>Affects Versions: 1.8.3
>Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Major
> Fix For: 1.8.3
>
>
> Please link to any issues that should be considered blockers for the 1.8.3 
> release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2163) Release 1.8.3

[Suraj Acharya (JIRA)]

Suraj Acharya created AVRO-2163:
---

 Summary: Release 1.8.3
 Key: AVRO-2163
 URL: https://issues.apache.org/jira/browse/AVRO-2163
 Project: Avro
  Issue Type: Task
  Components: community
Affects Versions: 1.8.2
Reporter: Sean Busbey
Assignee: Suraj Acharya
 Fix For: 1.8.2


Please link to any issues that should be considered blockers for the 1.8.2 
release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2163) Release 1.8.3

[Suraj Acharya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suraj Acharya updated AVRO-2163:

Affects Version/s: (was: 1.8.2)
   1.8.3

> Release 1.8.3
> -
>
> Key: AVRO-2163
> URL: https://issues.apache.org/jira/browse/AVRO-2163
> Project: Avro
>  Issue Type: Task
>  Components: community
>Affects Versions: 1.8.3
>Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Major
> Fix For: 1.8.3
>
>
> Please link to any issues that should be considered blockers for the 1.8.2 
> release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API

[Mike Yoder (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409861#comment-16409861
 ] 

Mike Yoder commented on AVRO-1605:
--

I wanted to chime in on this issue from a security perspective.  The TL;DR is 
that the use of jackson 1.x is dangerous.
 * The last version of jackson 1.x was 1.9.13, and that was released 5 years 
ago. The developers have moved to jackson 2.x and are no longer making patches 
for jackson 1.x.
 * A number of related security vulnerabilities have surfaced in jackson: 
CVE-2017-7525, CVE-2017-15095, CVE-2017-17485 and CVE-2018-5968.
 * In the wake of Equifax, many large organizations are taking the stance that 
"thou shalt not use third party libraries with security vulnerabilities".  

You can see where this takes us.

I don't really care what the solution is, but somehow Avro needs to move to 
jackson 2.x.  It would seem to me to be highly sensible to get jackson out of 
the Avro public interface now in order for this sort of issue to not happen in 
the future - but hey I'll take any solution I can get at this point.

 

> Remove Jackson classes from public API
> --
>
> Key: AVRO-1605
> URL: https://issues.apache.org/jira/browse/AVRO-1605
> Project: Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.7.8
>Reporter: Tom White
>Assignee: Gabor Szadovszky
>Priority: Major
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: WELCOME to dev@avro.apache.org

[Bridger Howell]

Not a committer, but I'd like to chime in and say that I'd absolutely love
to see another set of official releases soon.

When looking through the blockers in JIRA, AVRO-1810 and AVRO-1891 stand
out a bit - does anybody know if those issues should actually be considered
"blockers"?
Caveat here: I do have a personal interest in seeing AVRO-1891 fixed, and
if we do decide that it is actually a blocker, I would definitely be
willing to volunteer some time to help sort it out quickly.

On Thu, Mar 22, 2018 at 8:55 AM, Edward Anderson 
wrote:

> Hi,
>
> There have been a lot of great improvements
>  since May
> 2017 when Avro 1.8.2 was released—241 commits from 55 different
> contributors over nearly a year. We would love to see these included in a
> new versioned release. Right now, people affected by this year's worth of
> fixes need to work around the issues or do a local build of master, both of
> which are inconvenient. The number of users affected will continue to grow
> until the next release.
>
> Do you agree that it's time for a new release? If not now, when do you
> think will be best? We wanted to check in before starting our own
> workarounds.
>
> Thanks for all your work on this great project!
>
> Best,
>
> Edward Anderson
> Software Engineer
> Doximity, Inc.
>
> On Thu, Mar 22, 2018 at 10:54 AM,  wrote:
>
> > Hi! This is the ezmlm program. I'm managing the
> > dev@avro.apache.org mailing list.
> >
> > I'm working for my owner, who can be reached
> > at dev-ow...@avro.apache.org.
> >
> > Acknowledgment: I have added the address
> >
> >eander...@doximity.com
> >
> > to the dev mailing list.
> >
> > Welcome to dev@avro.apache.org!
> >
> > Please save this message so that you know the address you are
> > subscribed under, in case you later want to unsubscribe or change your
> > subscription address.
> >
> >
> > --- Administrative commands for the dev list ---
> >
> > I can handle administrative requests automatically. Please
> > do not send them to the list address! Instead, send
> > your message to the correct command address:
> >
> > To subscribe to the list, send a message to:
> >
> >
> > To remove your address from the list, send a message to:
> >
> >
> > Send mail to the following for info and FAQ for this list:
> >
> >
> >
> > Similar addresses exist for the digest list:
> >
> >
> >
> > To get messages 123 through 145 (a maximum of 100 per request), mail:
> >
> >
> > To get an index with subject and author for messages 123-456 , mail:
> >
> >
> > They are always returned as sets of 100, max 2000 per request,
> > so you'll actually get 100-499.
> >
> > To receive all messages with the same subject as message 12345,
> > send a short message to:
> >
> >
> > The messages should contain one line or word of text to avoid being
> > treated as sp@m, but I will ignore their content.
> > Only the ADDRESS you send to is important.
> >
> > You can start a subscription for an alternate address,
> > for example "john@host.domain", just add a hyphen and your
> > address (with '=' instead of '@') after the command word:
> > 

[jira] [Commented] (AVRO-2162) Add Zstandard compression to avro file format

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409699#comment-16409699
 ] 

Sean Busbey commented on AVRO-2162:
---

this would be great!

> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: WELCOME to dev@avro.apache.org

[Edward Anderson]

Hi,

There have been a lot of great improvements
 since May
2017 when Avro 1.8.2 was released—241 commits from 55 different
contributors over nearly a year. We would love to see these included in a
new versioned release. Right now, people affected by this year's worth of
fixes need to work around the issues or do a local build of master, both of
which are inconvenient. The number of users affected will continue to grow
until the next release.

Do you agree that it's time for a new release? If not now, when do you
think will be best? We wanted to check in before starting our own
workarounds.

Thanks for all your work on this great project!

Best,

Edward Anderson
Software Engineer
Doximity, Inc.

On Thu, Mar 22, 2018 at 10:54 AM,  wrote:

> Hi! This is the ezmlm program. I'm managing the
> dev@avro.apache.org mailing list.
>
> I'm working for my owner, who can be reached
> at dev-ow...@avro.apache.org.
>
> Acknowledgment: I have added the address
>
>eander...@doximity.com
>
> to the dev mailing list.
>
> Welcome to dev@avro.apache.org!
>
> Please save this message so that you know the address you are
> subscribed under, in case you later want to unsubscribe or change your
> subscription address.
>
>
> --- Administrative commands for the dev list ---
>
> I can handle administrative requests automatically. Please
> do not send them to the list address! Instead, send
> your message to the correct command address:
>
> To subscribe to the list, send a message to:
>
>
> To remove your address from the list, send a message to:
>
>
> Send mail to the following for info and FAQ for this list:
>
>
>
> Similar addresses exist for the digest list:
>
>
>
> To get messages 123 through 145 (a maximum of 100 per request), mail:
>
>
> To get an index with subject and author for messages 123-456 , mail:
>
>
> They are always returned as sets of 100, max 2000 per request,
> so you'll actually get 100-499.
>
> To receive all messages with the same subject as message 12345,
> send a short message to:
>
>
> The messages should contain one line or word of text to avoid being
> treated as sp@m, but I will ignore their content.
> Only the ADDRESS you send to is important.
>
> You can start a subscription for an alternate address,
> for example "john@host.domain", just add a hyphen and your
> address (with '=' instead of '@') after the command word:
> 

[jira] [Updated] (AVRO-2162) Add Zstandard compression to avro file format

[Scott Carey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Carey updated AVRO-2162:
--
Description: 
I'd like to add Zstandard compression for Avro. 

At compression level 1 It is almost as fast as Snappy at compression, with 
compression ratios more like gzip.  At higher levels of compression, it is more 
compact than gzip -9 with much lower CPU when compressing and roughly 3x faster 
decompression.

 

Adding it to Java is fairly easy.  We'll need to say something about it in the 
spec however, as an 'optinal' codec.

 

  was:
I'd like to add Zstandard compression for Avro. 

It is almost as fast as Snappy at compression, with compression ratios more 
like gzip.  At higher levels of compression, it is more compact than gzip -9 
with much lower CPU when compressing and roughly 3x faster decompression.

 

Adding it to Java is fairly easy.  We'll need to say something about it in the 
spec however, as an 'optinal' codec.

 


> Add Zstandard compression to avro file format
> -
>
> Key: AVRO-2162
> URL: https://issues.apache.org/jira/browse/AVRO-2162
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Scott Carey
>Priority: Major
>
> I'd like to add Zstandard compression for Avro. 
> At compression level 1 It is almost as fast as Snappy at compression, with 
> compression ratios more like gzip.  At higher levels of compression, it is 
> more compact than gzip -9 with much lower CPU when compressing and roughly 3x 
> faster decompression.
>  
> Adding it to Java is fairly easy.  We'll need to say something about it in 
> the spec however, as an 'optinal' codec.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2162) Add Zstandard compression to avro file format

[Scott Carey (JIRA)]

Scott Carey created AVRO-2162:
-

 Summary: Add Zstandard compression to avro file format
 Key: AVRO-2162
 URL: https://issues.apache.org/jira/browse/AVRO-2162
 Project: Avro
  Issue Type: Improvement
  Components: java
Reporter: Scott Carey


I'd like to add Zstandard compression for Avro. 

It is almost as fast as Snappy at compression, with compression ratios more 
like gzip.  At higher levels of compression, it is more compact than gzip -9 
with much lower CPU when compressing and roughly 3x faster decompression.

 

Adding it to Java is fairly easy.  We'll need to say something about it in the 
spec however, as an 'optinal' codec.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409170#comment-16409170
 ] 

ASF GitHub Bot commented on AVRO-2069:
--

scottcarey commented on issue #243: AVRO-2069: Use primitive fields in 
generated getters & setters in Java code
URL: https://github.com/apache/avro/pull/243#issuecomment-375204378
 
 
   This can be an option at generation time for those that don't want to break 
compatibility (defaulting to the old way prior to 1.9). 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use primitive fields in generated getters & setters in Java code
> 
>
> Key: AVRO-2069
> URL: https://issues.apache.org/jira/browse/AVRO-2069
> Project: Avro
>  Issue Type: Improvement
>Affects Versions: 1.8.2
>Reporter: Daniil Gitelson
>Assignee: Daniil Gitelson
>Priority: Major
>
> Currently, for primitive types (such as int, long, etc) generated getters and 
> setters return and accept java.lang.* boxed (while fields actually holds 
> primitive values). This is inefeccient and produces code boilerplate.
> Changed this behaviour in pull request: 
> https://github.com/apache/avro/pull/243



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)