date:20180509

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

2018-05-09 Thread Henry Robinson

+1 (non-binding)

I built Spark with the new artifacts, and confirmed that the test I wrote
for SPARK-23852 (the same issue as PARQUET-1217) passes.

Thanks for producing this release!

On 9 May 2018 at 08:51, Ryan Blue  wrote:

> Thanks, Gabor! Looks good to me.
>
> Changing my vote to +1.
>
> rb
>
> On Wed, May 9, 2018 at 5:20 AM, Gabor Szadovszky <
> gabor.szadovs...@cloudera.com> wrote:
>
> > Created PARQUET-1294 
> to
> > track it.
> >
> > Gabor
> >
> >
> > On 9 May 2018, at 13:42, Zoltan Ivanfi  wrote:
> >
> > Done. On the other hand, this is what the release scripts generate (with
> > the commit message "Apache Parquet MR $version RC${rc}" in which the
> > variables do not get expanded), so they will have to be updated as well.
> We
> > will add a JIRA for that.
> >
> > Zoltan
> >
> > On Tue, May 8, 2018 at 10:53 PM Ryan Blue 
> > wrote:
> >
> > +0
> >
> > The signature is good and I was able to build and test.
> >
> > The release doesn't conform to a recently-updated policy change for
> > checksums. Specifically, the sha file should be named sha1 but sha512 is
> > recommended, and there should not be a md5 checksum:
> > http://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> > Could you guys create a sha512 file and delete the other two checksums?
> > That would change my vote to a +1.
> >
> > rb
> >
> > On Tue, May 8, 2018 at 7:26 AM, Zoltan Ivanfi  wrote:
> >
> > +1 (binding)
> >
> > built and tested
> > verified signature
> >
> > I agree with Uwe that a verification script would be useful.
> >
> > Zoltan
> >
> > On Mon, May 7, 2018 at 5:37 PM Uwe L. Korn  wrote:
> >
> > +1 (binding)
> >
> > * Built and tested on Debian 8
> > * verified sha1
> > * verified signature
> >
> > was quite a hassle to build with manually installing protobuf and
> >
> > thrift.
> >
> > For newer releases, there definitely needs to be a verification script
> > otherwise voting is quite a labor intensive process.
> >
> > Uwe
> >
> > On Mon, May 7, 2018, at 9:58 AM, Gabor Szadovszky wrote:
> >
> > Hi Uwe,
> >
> > I guess this is because you are building it with java8. The 1.8.3
> >
> > branch
> >
> > is still on 1.6 (source and target) and travis is configured to use
> > jdk7. We also used jdk7 for the build.
> >
> > Cheers,
> > Gabor
> >
> > On 7 May 2018, at 09:46, Uwe L. Korn  wrote:
> >
> > Hello,
> >
> > the build is failing for me with "[ERROR] Failed to execute goal
> >
> > org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
> > (default) on project parquet-generator: Error rendering velocity
> >
> > resource.:
> >
> > NullPointerException", exteneded stacktrace:
> > https://gist.github.com/xhochy/fd62748ba8c300a5f238a80e8bacfc90
> >
> >
> > I can provide more information if you can tell me what you would
> >
> > need.
> >
> >
> > Uwe
> >
> > On Fri, May 4, 2018, at 2:12 PM, Gabor Szadovszky wrote:
> >
> > Hi everyone,
> >
> > Zoltan and I propose the following RC to be released as official
> >
> > Apache
> >
> > Parquet MR 1.8.3 release.
> >
> > The commit id is aef7230e114214b7cc962a8f3fc5aeed6ce80828
> > * This corresponds to the tag: apache-parquet-1.8.3
> > *
> >
> >
> > https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
> >
> > ed6ce80828
> >
> > <
> >
> >
> > https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
> >
> > ed6ce80828
> >
> >
> >
> > The release tarball, signature, and checksums are here:
> > *
> >
> >
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
> >
> > <
> >
> >
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
> >
> >
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> > 
> >
> > Binary artifacts are staged in Nexus here:
> > *
> >
> >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/
> >
> > <
> >
> >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/
> >
> >
> >
> > This is a maintenance release created mainly for Spark containing
> >
> > 2
> >
> > bug
> >
> > fixes related to the statistics handling.
> > See
> >
> >
> > https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
> >
> > ed6ce80828/CHANGES.md
> >
> > <
> >
> >
> > https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
> >
> > ed6ce80828/CHANGES.md>
> >
> >
> > for details.
> >
> > Please download, verify, and test.
> >
> > [ ] +1 Release this as Apache Parquet MR 1.8.3
> > [ ] +0
> > [ ] -1 Do not release this because…
> >
> >
> >
> >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
> >
> >
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

[jira] [Commented] (PARQUET-1195) org.apache.commons.codec.binary.Base64 clash #449

2018-05-09 Thread Brian McDevitt (JIRA)


[ 
https://issues.apache.org/jira/browse/PARQUET-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469601#comment-16469601
 ] 

Brian McDevitt commented on PARQUET-1195:
-

This looks like a duplicate of PARQUET-1194.

> org.apache.commons.codec.binary.Base64 clash #449
> -
>
> Key: PARQUET-1195
> URL: https://issues.apache.org/jira/browse/PARQUET-1195
> Project: Parquet
>  Issue Type: Bug
>  Components: parquet-mr
>Reporter: JaredLi
>Priority: Major
>
> The class "org.apache.commons.codec.binary.Base64" in parquet-tools.jar 
> version 1.9.0 is same at package and class name to which is in 
> commons-codec.jar(all version) ,but its implement is much older.
> That makes a jar-hell when both parquet-tools.jar and commons-codec.jar are 
> used .
> e.g.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PARQUET-1295) Parquet libraries do not follow proper semantic versioning

2018-05-09 Thread Vlad Rozov (JIRA)

Vlad Rozov created PARQUET-1295:
---

 Summary: Parquet libraries do not follow proper semantic versioning
 Key: PARQUET-1295
 URL: https://issues.apache.org/jira/browse/PARQUET-1295
 Project: Parquet
  Issue Type: Bug
Reporter: Vlad Rozov


There are changes between 1.8.0 and 1.10.0 that break API compatibility. A 
minor version change is supposed to be backward compatible with 1.9.0 and 1.8.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

2018-05-09 Thread Ryan Blue

Thanks, Gabor! Looks good to me.

Changing my vote to +1.

rb

On Wed, May 9, 2018 at 5:20 AM, Gabor Szadovszky <
gabor.szadovs...@cloudera.com> wrote:

> Created PARQUET-1294  to
> track it.
>
> Gabor
>
>
> On 9 May 2018, at 13:42, Zoltan Ivanfi  wrote:
>
> Done. On the other hand, this is what the release scripts generate (with
> the commit message "Apache Parquet MR $version RC${rc}" in which the
> variables do not get expanded), so they will have to be updated as well. We
> will add a JIRA for that.
>
> Zoltan
>
> On Tue, May 8, 2018 at 10:53 PM Ryan Blue 
> wrote:
>
> +0
>
> The signature is good and I was able to build and test.
>
> The release doesn't conform to a recently-updated policy change for
> checksums. Specifically, the sha file should be named sha1 but sha512 is
> recommended, and there should not be a md5 checksum:
> http://www.apache.org/dev/release-distribution#sigs-and-sums
>
> Could you guys create a sha512 file and delete the other two checksums?
> That would change my vote to a +1.
>
> rb
>
> On Tue, May 8, 2018 at 7:26 AM, Zoltan Ivanfi  wrote:
>
> +1 (binding)
>
> built and tested
> verified signature
>
> I agree with Uwe that a verification script would be useful.
>
> Zoltan
>
> On Mon, May 7, 2018 at 5:37 PM Uwe L. Korn  wrote:
>
> +1 (binding)
>
> * Built and tested on Debian 8
> * verified sha1
> * verified signature
>
> was quite a hassle to build with manually installing protobuf and
>
> thrift.
>
> For newer releases, there definitely needs to be a verification script
> otherwise voting is quite a labor intensive process.
>
> Uwe
>
> On Mon, May 7, 2018, at 9:58 AM, Gabor Szadovszky wrote:
>
> Hi Uwe,
>
> I guess this is because you are building it with java8. The 1.8.3
>
> branch
>
> is still on 1.6 (source and target) and travis is configured to use
> jdk7. We also used jdk7 for the build.
>
> Cheers,
> Gabor
>
> On 7 May 2018, at 09:46, Uwe L. Korn  wrote:
>
> Hello,
>
> the build is failing for me with "[ERROR] Failed to execute goal
>
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
> (default) on project parquet-generator: Error rendering velocity
>
> resource.:
>
> NullPointerException", exteneded stacktrace:
> https://gist.github.com/xhochy/fd62748ba8c300a5f238a80e8bacfc90
>
>
> I can provide more information if you can tell me what you would
>
> need.
>
>
> Uwe
>
> On Fri, May 4, 2018, at 2:12 PM, Gabor Szadovszky wrote:
>
> Hi everyone,
>
> Zoltan and I propose the following RC to be released as official
>
> Apache
>
> Parquet MR 1.8.3 release.
>
> The commit id is aef7230e114214b7cc962a8f3fc5aeed6ce80828
> * This corresponds to the tag: apache-parquet-1.8.3
> *
>
>
> https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
>
> ed6ce80828
>
> <
>
>
> https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
>
> ed6ce80828
>
>
>
> The release tarball, signature, and checksums are here:
> *
>
>
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
>
> <
>
>
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
>
>
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> 
>
> Binary artifacts are staged in Nexus here:
> *
>
>
> https://repository.apache.org/content/groups/staging/org/apache/parquet/
>
> <
>
>
> https://repository.apache.org/content/groups/staging/org/apache/parquet/
>
>
>
> This is a maintenance release created mainly for Spark containing
>
> 2
>
> bug
>
> fixes related to the statistics handling.
> See
>
>
> https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
>
> ed6ce80828/CHANGES.md
>
> <
>
>
> https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
>
> ed6ce80828/CHANGES.md>
>
>
> for details.
>
> Please download, verify, and test.
>
> [ ] +1 Release this as Apache Parquet MR 1.8.3
> [ ] +0
> [ ] -1 Do not release this because…
>
>
>
>
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>


-- 
Ryan Blue
Software Engineer
Netflix

[jira] [Commented] (PARQUET-1253) Support for new logical type representation

2018-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PARQUET-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468895#comment-16468895
 ] 

ASF GitHub Bot commented on PARQUET-1253:
-

gszadovszky commented on a change in pull request #463: PARQUET-1253: Support 
for new logical type representation
URL: https://github.com/apache/parquet-mr/pull/463#discussion_r187055416
 
 

 ##
 File path: 
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java
 ##
 @@ -36,42 +36,152 @@
 import org.apache.parquet.format.TimeType;
 import org.apache.parquet.format.TimestampType;
 
+import java.util.List;
 import java.util.Objects;
 
-public interface LogicalTypeAnnotation {
+public abstract class LogicalTypeAnnotation {
+  public enum LogicalTypes {
+MAP {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return mapType();
+  }
+},
+LIST {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return listType();
+  }
+},
+UTF8 {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return stringType();
+  }
+},
+MAP_KEY_VALUE {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return MapKeyValueTypeAnnotation.getInstance();
+  }
+},
+ENUM {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return enumType();
+  }
+},
+DECIMAL {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for decimal 
logical type, got " + params.size());
+}
+return decimalType(Integer.valueOf(params.get(1)), 
Integer.valueOf(params.get(0)));
+  }
+},
+DATE {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return dateType();
+  }
+},
+TIME {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for time logical 
type, got " + params.size());
+}
+return timeType(Boolean.parseBoolean(params.get(1)), 
TimeUnit.valueOf(params.get(0)));
+  }
+},
+TIMESTAMP {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for timestamp 
logical type, got " + params.size());
+}
+return timestampType(Boolean.parseBoolean(params.get(1)), 
TimeUnit.valueOf(params.get(0)));
+  }
+},
+INT {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for integer 
logical type, got " + params.size());
+}
+return intType(Integer.valueOf(params.get(0)), 
Boolean.parseBoolean(params.get(1)));
+  }
+},
+JSON {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return jsonType();
+  }
+},
+BSON {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return bsonType();
+  }
+},
+INTERVAL {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return IntervalLogicalTypeAnnotation.getInstance();
+  }
+};
+
+protected abstract LogicalTypeAnnotation fromString(List params);
+  }
+
   /**
* Convert this parquet-mr logical type to parquet-format LogicalType.
*
* @return the parquet-format LogicalType representation of this logical 
type implementation
*/
-  LogicalType toLogicalType();
+  public abstract LogicalType toLogicalType();
 
   /**
* Convert this parquet-mr logical type to parquet-format ConvertedType.
*
* @return the parquet-format ConvertedType representation of this logical 
type implementation
*/
-  ConvertedType toConvertedType();
+  public abstract ConvertedType toConvertedType();
 
   /**
* Convert this logical type to old logical type representation in 
parquet-mr (if there's any).
* Those logical type implementations, which don't have a corresponding 
mapping should return null.
*
* @return the OriginalType representation of the new logical type, or null 
if there's none
*/
-  OriginalType toOriginalType();
+  public abstract OriginalType toOriginalType();
 
   /**
* Visits this logical type with the given visitor
*
* @param logicalTypeAnnotationVisitor the visitor to visit this type
*/
-  void accept(LogicalTypeAnnotationVisitor logicalTypeAnnotationVisitor);
+  public abstract void accept(LogicalTypeAnnotationVisitor

[jira] [Commented] (PARQUET-1253) Support for new logical type representation

2018-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PARQUET-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468841#comment-16468841
 ] 

ASF GitHub Bot commented on PARQUET-1253:
-

nandorKollar commented on a change in pull request #463: PARQUET-1253: Support 
for new logical type representation
URL: https://github.com/apache/parquet-mr/pull/463#discussion_r187041286
 
 

 ##
 File path: 
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java
 ##
 @@ -36,42 +36,152 @@
 import org.apache.parquet.format.TimeType;
 import org.apache.parquet.format.TimestampType;
 
+import java.util.List;
 import java.util.Objects;
 
-public interface LogicalTypeAnnotation {
+public abstract class LogicalTypeAnnotation {
+  public enum LogicalTypes {
+MAP {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return mapType();
+  }
+},
+LIST {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return listType();
+  }
+},
+UTF8 {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return stringType();
+  }
+},
+MAP_KEY_VALUE {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return MapKeyValueTypeAnnotation.getInstance();
+  }
+},
+ENUM {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return enumType();
+  }
+},
+DECIMAL {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for decimal 
logical type, got " + params.size());
+}
+return decimalType(Integer.valueOf(params.get(1)), 
Integer.valueOf(params.get(0)));
+  }
+},
+DATE {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return dateType();
+  }
+},
+TIME {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for time logical 
type, got " + params.size());
+}
+return timeType(Boolean.parseBoolean(params.get(1)), 
TimeUnit.valueOf(params.get(0)));
+  }
+},
+TIMESTAMP {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for timestamp 
logical type, got " + params.size());
+}
+return timestampType(Boolean.parseBoolean(params.get(1)), 
TimeUnit.valueOf(params.get(0)));
+  }
+},
+INT {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+if (params.size() != 2) {
+  throw new RuntimeException("Expecting 2 parameters for integer 
logical type, got " + params.size());
+}
+return intType(Integer.valueOf(params.get(0)), 
Boolean.parseBoolean(params.get(1)));
+  }
+},
+JSON {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return jsonType();
+  }
+},
+BSON {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return bsonType();
+  }
+},
+INTERVAL {
+  @Override
+  protected LogicalTypeAnnotation fromString(List params) {
+return IntervalLogicalTypeAnnotation.getInstance();
+  }
+};
+
+protected abstract LogicalTypeAnnotation fromString(List params);
+  }
+
   /**
* Convert this parquet-mr logical type to parquet-format LogicalType.
*
* @return the parquet-format LogicalType representation of this logical 
type implementation
*/
-  LogicalType toLogicalType();
+  public abstract LogicalType toLogicalType();
 
   /**
* Convert this parquet-mr logical type to parquet-format ConvertedType.
*
* @return the parquet-format ConvertedType representation of this logical 
type implementation
*/
-  ConvertedType toConvertedType();
+  public abstract ConvertedType toConvertedType();
 
   /**
* Convert this logical type to old logical type representation in 
parquet-mr (if there's any).
* Those logical type implementations, which don't have a corresponding 
mapping should return null.
*
* @return the OriginalType representation of the new logical type, or null 
if there's none
*/
-  OriginalType toOriginalType();
+  public abstract OriginalType toOriginalType();
 
   /**
* Visits this logical type with the given visitor
*
* @param logicalTypeAnnotationVisitor the visitor to visit this type
*/
-  void accept(LogicalTypeAnnotationVisitor logicalTypeAnnotationVisitor);
+  public abstract void accept(LogicalTypeAnnotationVisitor

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

2018-05-09 Thread Gabor Szadovszky

Created PARQUET-1294  to 
track it.

Gabor

> On 9 May 2018, at 13:42, Zoltan Ivanfi  wrote:
> 
> Done. On the other hand, this is what the release scripts generate (with
> the commit message "Apache Parquet MR $version RC${rc}" in which the
> variables do not get expanded), so they will have to be updated as well. We
> will add a JIRA for that.
> 
> Zoltan
> 
> On Tue, May 8, 2018 at 10:53 PM Ryan Blue  wrote:
> 
>> +0
>> 
>> The signature is good and I was able to build and test.
>> 
>> The release doesn't conform to a recently-updated policy change for
>> checksums. Specifically, the sha file should be named sha1 but sha512 is
>> recommended, and there should not be a md5 checksum:
>> http://www.apache.org/dev/release-distribution#sigs-and-sums
>> 
>> Could you guys create a sha512 file and delete the other two checksums?
>> That would change my vote to a +1.
>> 
>> rb
>> 
>> On Tue, May 8, 2018 at 7:26 AM, Zoltan Ivanfi  wrote:
>> 
>>> +1 (binding)
>>> 
>>> built and tested
>>> verified signature
>>> 
>>> I agree with Uwe that a verification script would be useful.
>>> 
>>> Zoltan
>>> 
>>> On Mon, May 7, 2018 at 5:37 PM Uwe L. Korn  wrote:
>>> 
 +1 (binding)
 
 * Built and tested on Debian 8
 * verified sha1
 * verified signature
 
 was quite a hassle to build with manually installing protobuf and
>> thrift.
 For newer releases, there definitely needs to be a verification script
 otherwise voting is quite a labor intensive process.
 
 Uwe
 
 On Mon, May 7, 2018, at 9:58 AM, Gabor Szadovszky wrote:
> Hi Uwe,
> 
> I guess this is because you are building it with java8. The 1.8.3
>>> branch
> is still on 1.6 (source and target) and travis is configured to use
> jdk7. We also used jdk7 for the build.
> 
> Cheers,
> Gabor
> 
>> On 7 May 2018, at 09:46, Uwe L. Korn  wrote:
>> 
>> Hello,
>> 
>> the build is failing for me with "[ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
 (default) on project parquet-generator: Error rendering velocity
>>> resource.:
 NullPointerException", exteneded stacktrace:
 https://gist.github.com/xhochy/fd62748ba8c300a5f238a80e8bacfc90
>> 
>> I can provide more information if you can tell me what you would
>>> need.
>> 
>> Uwe
>> 
>> On Fri, May 4, 2018, at 2:12 PM, Gabor Szadovszky wrote:
>>> Hi everyone,
>>> 
>>> Zoltan and I propose the following RC to be released as official
 Apache
>>> Parquet MR 1.8.3 release.
>>> 
>>> The commit id is aef7230e114214b7cc962a8f3fc5aeed6ce80828
>>> * This corresponds to the tag: apache-parquet-1.8.3
>>> *
>>> 
 
>> https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
>>> ed6ce80828
>>> <
 
>> https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
>>> ed6ce80828
> 
>>> 
>>> The release tarball, signature, and checksums are here:
>>> *
>>> 
 
>> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
>>> <
 
>> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
 
>>> 
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
>>> 
>>> 
>>> Binary artifacts are staged in Nexus here:
>>> *
>>> 
 
>> https://repository.apache.org/content/groups/staging/org/apache/parquet/
>>> <
 
>> https://repository.apache.org/content/groups/staging/org/apache/parquet/
 
>>> 
>>> This is a maintenance release created mainly for Spark containing
>> 2
 bug
>>> fixes related to the statistics handling.
>>> See
>>> 
 
>> https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
>>> ed6ce80828/CHANGES.md
>>> <
 
>> https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
>>> ed6ce80828/CHANGES.md>
 
>>> for details.
>>> 
>>> Please download, verify, and test.
>>> 
>>> [ ] +1 Release this as Apache Parquet MR 1.8.3
>>> [ ] +0
>>> [ ] -1 Do not release this because…
> 
 
>>> 
>> 
>> 
>> 
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

2018-05-09 Thread Zoltan Ivanfi

Done. On the other hand, this is what the release scripts generate (with
the commit message "Apache Parquet MR $version RC${rc}" in which the
variables do not get expanded), so they will have to be updated as well. We
will add a JIRA for that.

Zoltan

On Tue, May 8, 2018 at 10:53 PM Ryan Blue  wrote:

> +0
>
> The signature is good and I was able to build and test.
>
> The release doesn't conform to a recently-updated policy change for
> checksums. Specifically, the sha file should be named sha1 but sha512 is
> recommended, and there should not be a md5 checksum:
> http://www.apache.org/dev/release-distribution#sigs-and-sums
>
> Could you guys create a sha512 file and delete the other two checksums?
> That would change my vote to a +1.
>
> rb
>
> On Tue, May 8, 2018 at 7:26 AM, Zoltan Ivanfi  wrote:
>
> > +1 (binding)
> >
> > built and tested
> > verified signature
> >
> > I agree with Uwe that a verification script would be useful.
> >
> > Zoltan
> >
> > On Mon, May 7, 2018 at 5:37 PM Uwe L. Korn  wrote:
> >
> > > +1 (binding)
> > >
> > > * Built and tested on Debian 8
> > > * verified sha1
> > > * verified signature
> > >
> > > was quite a hassle to build with manually installing protobuf and
> thrift.
> > > For newer releases, there definitely needs to be a verification script
> > > otherwise voting is quite a labor intensive process.
> > >
> > > Uwe
> > >
> > > On Mon, May 7, 2018, at 9:58 AM, Gabor Szadovszky wrote:
> > > > Hi Uwe,
> > > >
> > > > I guess this is because you are building it with java8. The 1.8.3
> > branch
> > > > is still on 1.6 (source and target) and travis is configured to use
> > > > jdk7. We also used jdk7 for the build.
> > > >
> > > > Cheers,
> > > > Gabor
> > > >
> > > > > On 7 May 2018, at 09:46, Uwe L. Korn  wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > the build is failing for me with "[ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
> > > (default) on project parquet-generator: Error rendering velocity
> > resource.:
> > > NullPointerException", exteneded stacktrace:
> > > https://gist.github.com/xhochy/fd62748ba8c300a5f238a80e8bacfc90
> > > > >
> > > > > I can provide more information if you can tell me what you would
> > need.
> > > > >
> > > > > Uwe
> > > > >
> > > > > On Fri, May 4, 2018, at 2:12 PM, Gabor Szadovszky wrote:
> > > > >> Hi everyone,
> > > > >>
> > > > >> Zoltan and I propose the following RC to be released as official
> > > Apache
> > > > >> Parquet MR 1.8.3 release.
> > > > >>
> > > > >> The commit id is aef7230e114214b7cc962a8f3fc5aeed6ce80828
> > > > >> * This corresponds to the tag: apache-parquet-1.8.3
> > > > >> *
> > > > >>
> > >
> https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
> > ed6ce80828
> > > > >> <
> > >
> https://github.com/apache/parquet-mr/tree/aef7230e114214b7cc962a8f3fc5ae
> > ed6ce80828
> > > >
> > > > >>
> > > > >> The release tarball, signature, and checksums are here:
> > > > >> *
> > > > >>
> > >
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
> > > > >> <
> > >
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.8.3-rc0/
> > > > >>>
> > > > >>
> > > > >> You can find the KEYS file here:
> > > > >> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> > > > >> 
> > > > >>
> > > > >> Binary artifacts are staged in Nexus here:
> > > > >> *
> > > > >>
> > >
> https://repository.apache.org/content/groups/staging/org/apache/parquet/
> > > > >> <
> > >
> https://repository.apache.org/content/groups/staging/org/apache/parquet/
> > > > >>>
> > > > >>
> > > > >> This is a maintenance release created mainly for Spark containing
> 2
> > > bug
> > > > >> fixes related to the statistics handling.
> > > > >> See
> > > > >>
> > >
> https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
> > ed6ce80828/CHANGES.md
> > > > >> <
> > >
> https://github.com/apache/parquet-mr/blob/aef7230e114214b7cc962a8f3fc5ae
> > ed6ce80828/CHANGES.md>
> > >
> > > > >> for details.
> > > > >>
> > > > >> Please download, verify, and test.
> > > > >>
> > > > >> [ ] +1 Release this as Apache Parquet MR 1.8.3
> > > > >> [ ] +0
> > > > >> [ ] -1 Do not release this because…
> > > >
> > >
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

[jira] [Created] (PARQUET-1294) Update release scripts for the new Apache policy

2018-05-09 Thread Gabor Szadovszky (JIRA)

Gabor Szadovszky created PARQUET-1294:
-

 Summary: Update release scripts for the new Apache policy
 Key: PARQUET-1294
 URL: https://issues.apache.org/jira/browse/PARQUET-1294
 Project: Parquet
  Issue Type: Task
  Components: parquet-format, parquet-mr
Reporter: Gabor Szadovszky


The Apache policy about the checksums is changed recently so it is required to 
update to related release scripts. See the policy detailed here: 
http://www.apache.org/dev/release-distribution#sigs-and-sums




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PARQUET-1293) Build failure when using Java 8 lambda expressions

2018-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PARQUET-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468656#comment-16468656
 ] 

ASF GitHub Bot commented on PARQUET-1293:
-

nandorKollar opened a new pull request #474: PARQUET-1293: Build failure when 
using Java 8 lambda expressions
URL: https://github.com/apache/parquet-mr/pull/474
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Build failure when using Java 8 lambda expressions
> --
>
> Key: PARQUET-1293
> URL: https://issues.apache.org/jira/browse/PARQUET-1293
> Project: Parquet
>  Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
>Priority: Minor
>
> I tried to use Java 8 lambda expressions, but when I tried {{mvn clean 
> install}}, I ran into compilation error:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:2.1:shade (default) on project 
> parquet-column: Error creating shaded jar: 10969 -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-shade-plugin:2.1:shade (default) on 
> project parquet-column: Error creating shaded jar: 10969
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>  at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error creating 
> shaded jar: 10969
>  at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:528)
>  at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>  at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>  ... 20 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 10969
>  at org.objectweb.asm.ClassReader.(Unknown Source)
>  at org.objectweb.asm.ClassReader.(Unknown Source)
>  at org.objectweb.asm.ClassReader.(Unknown Source)
>  at org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:94)
>  at 
> org.apache.maven.plugins.shade.filter.MinijarFilter.(MinijarFilter.java:77)
>  at 
> org.apache.maven.plugins.shade.mojo.ShadeMojo.getFilters(ShadeMojo.java:767)
>  at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:445)
>  ... 22 more
> {code}
>  
> Since Parquet is already on Java 8, one should be able to use all Java 8 
> features. Upgrading maven-shade-plugin version and enforcer-rule dependency 
> of 
> maven-enforcer-plugin would solve the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PARQUET-1293) Build failure when using Java 8 lambda expressions

2018-05-09 Thread Nandor Kollar (JIRA)

Nandor Kollar created PARQUET-1293:
--

 Summary: Build failure when using Java 8 lambda expressions
 Key: PARQUET-1293
 URL: https://issues.apache.org/jira/browse/PARQUET-1293
 Project: Parquet
  Issue Type: Bug
Affects Versions: 1.8.2
Reporter: Nandor Kollar
Assignee: Nandor Kollar


I tried to use Java 8 lambda expressions, but when I tried {{mvn clean 
install}}, I ran into compilation error:

{code}

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-shade-plugin:2.1:shade (default) on project 
parquet-column: Error creating shaded jar: 10969 -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-shade-plugin:2.1:shade (default) on project 
parquet-column: Error creating shaded jar: 10969

 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)

 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)

 at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)

 at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)

 at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)

 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)

 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)

 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)

 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)

 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 at java.lang.reflect.Method.invoke(Method.java:498)

 at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)

 at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)

 at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)

 at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: org.apache.maven.plugin.MojoExecutionException: Error creating 
shaded jar: 10969

 at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:528)

 at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)

 at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)

 ... 20 more

Caused by: java.lang.ArrayIndexOutOfBoundsException: 10969

 at org.objectweb.asm.ClassReader.(Unknown Source)

 at org.objectweb.asm.ClassReader.(Unknown Source)

 at org.objectweb.asm.ClassReader.(Unknown Source)

 at org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:94)

 at 
org.apache.maven.plugins.shade.filter.MinijarFilter.(MinijarFilter.java:77)

 at org.apache.maven.plugins.shade.mojo.ShadeMojo.getFilters(ShadeMojo.java:767)

 at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:445)

 ... 22 more

{code}

 

Since Parquet is already on Java 8, one should be able to use all Java 8 
features. Upgrading maven-shade-plugin version and enforcer-rule dependency of 

maven-enforcer-plugin would solve the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PARQUET-1253) Support for new logical type representation

2018-05-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/PARQUET-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468593#comment-16468593
 ] 

ASF GitHub Bot commented on PARQUET-1253:
-

nandorKollar commented on issue #463: PARQUET-1253: Support for new logical 
type representation
URL: https://github.com/apache/parquet-mr/pull/463#issuecomment-387680018
 
 
   @gszadovszky it looks like the version of maven-shade-plugin and 
enforcer-rule dependency in maven-enforcer-plugin Maven plugins used in Parquet 
don't like Java 8 lambda expressions, that's why Travis build failed. Since 
Parquet is already upgraded to Java 8, I think we should upgrade these plugins. 
I'll create a separate Jira for this issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Support for new logical type representation
> ---
>
> Key: PARQUET-1253
> URL: https://issues.apache.org/jira/browse/PARQUET-1253
> Project: Parquet
>  Issue Type: Improvement
>  Components: parquet-mr
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
>Priority: Major
>
> Latest parquet-format 
> [introduced|https://github.com/apache/parquet-format/commit/863875e0be3237c6aa4ed71733d54c91a51deabe#diff-0f9d1b5347959e15259da7ba8f4b6252]
>  a new representation for logical types. As of now this is not yet supported 
> in parquet-mr, thus there's no way to use parametrized UTC normalized 
> timestamp data types. When reading and writing Parquet files, besides 
> 'converted_type' parquet-mr should use the new 'logicalType' field in 
> SchemaElement to tell the current logical type annotation. To maintain 
> backward compatibility, the semantic of converted_type shouldn't change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

[jira] [Commented] (PARQUET-1195) org.apache.commons.codec.binary.Base64 clash #449

[jira] [Created] (PARQUET-1295) Parquet libraries do not follow proper semantic versioning

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

[jira] [Commented] (PARQUET-1253) Support for new logical type representation

[jira] [Commented] (PARQUET-1253) Support for new logical type representation

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

Re: [VOTE] Release Apache Parquet MR 1.8.3 RC0

[jira] [Created] (PARQUET-1294) Update release scripts for the new Apache policy

[jira] [Commented] (PARQUET-1293) Build failure when using Java 8 lambda expressions

[jira] [Created] (PARQUET-1293) Build failure when using Java 8 lambda expressions

[jira] [Commented] (PARQUET-1253) Support for new logical type representation

12 matches

Site Navigation

Mail list logo

Footer information