Test failures from flaky test
There's a flaky test in TestMemoryManager that is causing test failures for all of the new PRs. I fixed the problem, but it's still waiting for a +1 from a committer. (Thanks to Sergio for reviewing!) Could someone take a quick look so we can get CI back on track? Thanks! rb -- Ryan Blue
[jira] [Commented] (PARQUET-349) VersionParser does not handle versions like "parquet-mr 1.6.0rc4"
[ https://issues.apache.org/jira/browse/PARQUET-349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963892#comment-14963892 ] Reuben Kuhnert commented on PARQUET-349: Added patch (PR-283). > VersionParser does not handle versions like "parquet-mr 1.6.0rc4" > - > > Key: PARQUET-349 > URL: https://issues.apache.org/jira/browse/PARQUET-349 > Project: Parquet > Issue Type: Bug > Components: parquet-mr >Reporter: Alex Levenson >Assignee: Reuben Kuhnert >Priority: Minor > > I'm not sure how we missed this one (I thought I added a test for this but > apparently not :p) but the tests don't cover it, and the parser can't parse > it even though the intention was to support a missing (build abcd) section in > the version string. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Error Compiling Parquet 1.8.2-SNAPSHOT
Can anyone please help with the following ?
Not sure if I have done something wrong here.
downloaded zip from https://github.com/apache/parquet-mr/archive/master.zip
mvn -U clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.parquet:parquet-scala_2.10:jar:1.8.2-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.parquet:parquet-scala_${scala.binary.version}:[unknown-version],
/Users/cmathews/PROJECTS/HADOOP/parquet-mr-master/parquet-scala/pom.xml, line
29, column 15
[WARNING] 'build.plugins.plugin.version' for org.scala-tools:maven-scala-plugin
is missing. @
org.apache.parquet:parquet-scala_${scala.binary.version}:[unknown-version],
/Users/cmathews/PROJECTS/HADOOP/parquet-mr-master/parquet-scala/pom.xml, line
71, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.parquet:parquet-scrooge_2.10:jar:1.8.2-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.parquet:parquet-scrooge_${scala.binary.version}:[unknown-version],
/Users/cmathews/PROJECTS/HADOOP/parquet-mr-master/parquet-scrooge/pom.xml, line
29, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten
the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
[INFO]
[INFO] Reactor Build Order:
[INFO]
[INFO] Apache Parquet MR
[INFO] Apache Parquet Generator
[INFO] Apache Parquet Common
[INFO] Apache Parquet Encodings
[INFO] Apache Parquet Column
[INFO] Apache Parquet Jackson
[INFO] Apache Parquet Hadoop
[INFO] Apache Parquet Avro
[INFO] Apache Parquet Benchmarks
[INFO] Apache Parquet Pig
[INFO] Apache Parquet Thrift
[INFO] Apache Parquet Cascading
[INFO] Apache Parquet Pig Bundle
[INFO] Apache Parquet Protobuf
[INFO] Apache Parquet Scala
[INFO] Apache Parquet Scrooge
[INFO] Apache Parquet Hadoop Bundle
[INFO] Apache Parquet Hive
[INFO] Apache Parquet Hive Binding Parent
[INFO] Apache Parquet Hive Binding Interface
[INFO] Apache Parquet Hive 0.10 Binding
[INFO] Apache Parquet Hive 0.12 Binding
[INFO] Apache Parquet Hive Binding Factory
[INFO] Apache Parquet Hive Binding Bundle
[INFO] Apache Parquet Hive Storage Handler
[INFO] Apache Parquet Hive Bundle
[INFO] Apache Parquet Tools
[INFO]
[INFO]
[INFO] Building Apache Parquet MR 1.8.2-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ parquet ---
[INFO] Deleting /Users/cmathews/PROJECTS/HADOOP/parquet-mr-master/target
[INFO]
[INFO] --- buildnumber-maven-plugin:1.1:create (default) @ parquet ---
[INFO] Checking for local modifications: skipped.
[INFO] Updating project files from SCM: skipped.
[INFO] Executing: /bin/sh -c cd
/Users/cmathews/PROJECTS/HADOOP/parquet-mr-master && git rev-parse --verify HEAD
[INFO] Working directory: /Users/cmathews/PROJECTS/HADOOP/parquet-mr-master
[INFO] Storing buildNumber: null at timestamp: 1445276335909
[INFO] Executing: /bin/sh -c cd
/Users/cmathews/PROJECTS/HADOOP/parquet-mr-master && git rev-parse --verify HEAD
[INFO] Working directory: /Users/cmathews/PROJECTS/HADOOP/parquet-mr-master
[INFO] Storing buildScmBranch: UNKNOWN_BRANCH
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ parquet ---
[INFO]
[INFO] --- apache-rat-plugin:0.11:check (default) @ parquet ---
[INFO] 70 implicit excludes (use -debug for more details).
[INFO] Exclude: **/*.parquet
[INFO] Exclude: **/*.avro
[INFO] Exclude: **/*.json
[INFO] Exclude: **/names.txt
[INFO] Exclude: **/*.avsc
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.log
[INFO] Exclude: **/*.md.vm
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.project
[INFO] Exclude: **/.settings/**
[INFO] Exclude: **/build/**
[INFO] Exclude: **/target/**
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: .idea/**
[INFO] Exclude: */jdiff/*.xml
[INFO] Exclude: .travis.yml
[INFO] Exclude: licenses/**
[INFO] Exclude: protobuf_install/**
[INFO] Exclude: thrift-0.7.0/**
[INFO] Exclude: thrift-0.7.0.tar.gz
[INFO] Exclude: **/dependency-reduced-pom.xml
[INFO] 17 resources included (use -debug for more details)
Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
recognized.
Compiler warnings:
WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
Warning: org.apache.xerces.parsers.SAXParser: Feature
[jira] [Commented] (PARQUET-124) parquet.hadoop.ParquetOutputCommitter.commitJob() throws parquet.io.ParquetEncodingException
[
https://issues.apache.org/jira/browse/PARQUET-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964192#comment-14964192
]
Alex Nastetsky commented on PARQUET-124:
Still seeing this issue, at least when running locally. I think it needs to do
something like FileSystem.get(new URI("file:///"), new
Configuration()).resolvePath(X) on both "root" and "new Path(footerPath)",
except using the correct file system.
> parquet.hadoop.ParquetOutputCommitter.commitJob() throws
> parquet.io.ParquetEncodingException
>
>
> Key: PARQUET-124
> URL: https://issues.apache.org/jira/browse/PARQUET-124
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
>Affects Versions: 1.6.0
>Reporter: Chris Albright
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: PARQUET-124-test
>
>
> I'm running an example combining Avro, Spark and Parquet
> (https://github.com/massie/spark-parquet-example), and in the process of
> updating the library versions, am getting the warning below.
> The version of Parquet-Hadoop in the original example is 1.0.0. I am using
> 1.6.0rc3
> The ParquetFileWriter.mergeFooters(Path, List) method is performing a
> check to ensure the footers are all for files in the output directory. The
> output directory is supplied by ParquetFileWriter.writeMetadataFile; in
> 1.0.0, the output path was converted to a fully qualified output path before
> the call to mergeFooters, but in 1.6.0rc[2,3] that conversion happens after
> the call to mergeFooters. Because of this, the check within merge footers is
> failing (the URI for the footers starts with file:, but not the URI for the
> root path does not)
> Here is the warning message and stacktrace.
> Oct 30, 2014 9:11:31 PM WARNING: parquet.hadoop.ParquetOutputCommitter: could
> not write summary file for /tmp/1414728690018-0/output
> parquet.io.ParquetEncodingException:
> file:/tmp/1414728690018-0/output/part-r-0.parquet invalid: all the files
> must be contained in the root /tmp/1414728690018-0/output
> at
> parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:422)
> at
> parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:398)
> at
> parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:50)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:936)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> at
> com.zenfractal.SparkParquetExample$.main(SparkParquetExample.scala:72)
> at com.zenfractal.SparkParquetExample.main(SparkParquetExample.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (PARQUET-373) MemoryManager tests are flaky
[ https://issues.apache.org/jira/browse/PARQUET-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved PARQUET-373. --- Resolution: Fixed > MemoryManager tests are flaky > - > > Key: PARQUET-373 > URL: https://issues.apache.org/jira/browse/PARQUET-373 > Project: Parquet > Issue Type: Bug > Components: parquet-mr >Affects Versions: 1.8.0 >Reporter: Ryan Blue >Assignee: Ryan Blue > Fix For: 1.9.0 > > > The memory manager tests are flaky, depending on the heap allocation for the > JVM they run in. This is caused by over-specific tests that assert the memory > allocation down to the byte and the fact that some assertions implicitly cast > long values to doubles to use the "within" form of assertEquals. > The tests should not validate a specific allocation strategy, but should > instead assert that: > 1. The allocation for a file is the row group size until room runs out > 2. When scaling row groups, the total allocation does not exceed the pool size -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PARQUET-349) VersionParser does not handle versions like "parquet-mr 1.6.0rc4"
[ https://issues.apache.org/jira/browse/PARQUET-349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuben Kuhnert reassigned PARQUET-349: -- Assignee: Reuben Kuhnert > VersionParser does not handle versions like "parquet-mr 1.6.0rc4" > - > > Key: PARQUET-349 > URL: https://issues.apache.org/jira/browse/PARQUET-349 > Project: Parquet > Issue Type: Bug > Components: parquet-mr >Reporter: Alex Levenson >Assignee: Reuben Kuhnert >Priority: Minor > > I'm not sure how we missed this one (I thought I added a test for this but > apparently not :p) but the tests don't cover it, and the parser can't parse > it even though the intention was to support a missing (build abcd) section in > the version string. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
