[jira] [Resolved] (PARQUET-1674) The announcement email on the web site does not comply with ASF rules
[
https://issues.apache.org/jira/browse/PARQUET-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky resolved PARQUET-1674.
---
Resolution: Fixed
> The announcement email on the web site does not comply with ASF rules
> -
>
> Key: PARQUET-1674
> URL: https://issues.apache.org/jira/browse/PARQUET-1674
> Project: Parquet
> Issue Type: Bug
> Components: parquet-site
>Reporter: Jim Apple
>Assignee: Gabor Szadovszky
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 1h
> Time Spent: 20m
> Remaining Estimate: 40m
>
> After following the instructions in the release guide on the Parquet website,
> my mail to [annou...@apache.org|mailto:annou...@apache.org] was rejected with
> the following message:
>
> {quote}Sorry, but the announce email cannot be accepted as it stands.
> Announcements of Apache project releases must contain a link to the
> relevant download page. [1]
> The download page must provide public download links where current official
> source releases and accompanying cryptographic files may be obtained. [2]
> It must also link to the KEYS file at
> [https://www.apache.org/dist/]/KEYS,
> and provide details of how to verify a download using the signature or a
> hash [3]
> Note also that MD5 and SHA1 hashes are deprecated and should not be used
> for new releases. [4]
> Announcements that contain a link to the dyn/closer page alone will be
> rejected by the moderators.
> Announcements that contain a link to the
> [dist.apache.org|http://dist.apache.org/] host will be
> rejected by the moderators.
> Announcements that contain a link to a web page that does not include a
> link to a mirror to the artifact plus links to the signature and at least
> one sha checksum will be rejected.
> [1] [https://www.apache.org/legal/release-policy.html#release-announcements]
> <[http://www.apache.org/legal/release-policy.html#release-announcements]>
> [2] [https://www.apache.org/dev/release-distribution#download-links]
> [3] [https://www.apache.org/dev/release-download-pages.html#download-page]
> [4] [https://www.apache.org/dev/release-distribution#sigs-and-sums]
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1691) Build fails due to missing hadoop-lzo
[
https://issues.apache.org/jira/browse/PARQUET-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1691:
--
Fix Version/s: 1.11.0
> Build fails due to missing hadoop-lzo
> -
>
> Key: PARQUET-1691
> URL: https://issues.apache.org/jira/browse/PARQUET-1691
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
>Affects Versions: 1.11.0
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Travis builds fail due to missing
> {{com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16}}. See details in the
> [Travis log|https://api.travis-ci.org/v3/job/608466783/log.txt]. Issue is
> reproducible locally be removing {{hadoop-lzo}} from the local maven
> repository and starting the build.
> See more details about the issue in the [hadoop-lzo bug
> tracking|https://github.com/twitter/hadoop-lzo/issues/140].
> {{hadoop-lzo}} is not required for build/test parquet-mr; it is a transitive
> dependency pulled in by {{elephant-bird-core}}. Possible fix is to exclude
> {{hadoop-lzo}} from the dependency tree.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1351) Travis builds fail for parquet-format
[
https://issues.apache.org/jira/browse/PARQUET-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1351:
--
Fix Version/s: (was: 1.11.0)
format-2.6.0
> Travis builds fail for parquet-format
> -
>
> Key: PARQUET-1351
> URL: https://issues.apache.org/jira/browse/PARQUET-1351
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Nándor Kollár
>Assignee: Nándor Kollár
>Priority: Major
> Labels: pull-request-available
> Fix For: format-2.6.0
>
>
> parquet-format Travis builds are failing with (see for example
> [this|https://travis-ci.org/apache/parquet-format/builds/400272129?utm_source=github_status_medium=notification]
> build):
> {code}
> [ERROR] Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or
> one of its dependencies could not be resolved: Failed to read artifact
> descriptor for
> org.apache.maven.plugins:maven-remote-resources-plugin:jar:1.5: Could not
> transfer artifact
> org.apache.maven.plugins:maven-remote-resources-plugin:pom:1.5 from/to
> central (https://repo.maven.apache.org/maven2): Received fatal alert:
> protocol_version -> [Help 1]
> {code}
> After some investigation, I suspect that the problem is that parquet-format
> is built against JDK 1.7.0_80 (see in the logs of the failing job above), the
> default version of TLS for this JDK seems to be TLSv1 (see
> [here|https://blogs.oracle.com/java-platform-group/diagnosing-tls,-ssl,-and-https]),
> and Maven central dropped support TLS 1.1 and TLS 1.0 (see
> [here|https://lists.apache.org/thread.html/%3CCAMk_Beg9=QE-UB10c=yM8NKWiAk=bak2n3da0pgyxdfhf3w...@mail.gmail.com%3E]
> and
> [here|https://stackoverflow.com/questions/50824789/why-am-i-getting-received-fatal-alert-protocol-version-or-peer-not-authentic]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1687) Update release process
[
https://issues.apache.org/jira/browse/PARQUET-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1687:
--
Issue Type: Task (was: Improvement)
> Update release process
> --
>
> Key: PARQUET-1687
> URL: https://issues.apache.org/jira/browse/PARQUET-1687
> Project: Parquet
> Issue Type: Task
> Components: parquet-format, parquet-mr, parquet-site
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Our current tagging policy in the release process requires to use the same
> tag for all the release candidates which means at RC2 we remove the tag from
> RC1 head and adds again to the RC2 head and so on. It is not a good practice.
> Hard to track RCs and rewriting git history is usually not a good idea.
> Update the related scripts and docs so we will create RC tags (e.g.
> {{apache-parquet-1.11.0-rc6}}) first and add the final release tag (e.g.
> {{apache-parquet-1.11.0}}) after the vote passes.
> It is also very confusing that we always use SNAPSHOT version of the next
> release even if the previous one is not released yet. For example we have not
> released {{1.11.0}} yet (we are after the failed vote of 1.11.0-rc6) still,
> we use the version number {{1.12.0-SNAPSHOT}}. This is very confusing.
> Update the release scripts and docs to use the same {{SNAPSHOT}} version at
> RC phase and only step to the next version number after the vote passed and
> the final release tag is added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1551) Support Java 11 - top-level JIRA
[ https://issues.apache.org/jira/browse/PARQUET-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated PARQUET-1551: -- Fix Version/s: 1.11.0 > Support Java 11 - top-level JIRA > > > Key: PARQUET-1551 > URL: https://issues.apache.org/jira/browse/PARQUET-1551 > Project: Parquet > Issue Type: Task >Reporter: Zoltan Ivanfi >Priority: Major > Fix For: 1.11.0 > > > This JIRA groups all other JIRA-s related to Java 11. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (PARQUET-1551) Support Java 11 - top-level JIRA
[ https://issues.apache.org/jira/browse/PARQUET-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky resolved PARQUET-1551. --- Resolution: Fixed > Support Java 11 - top-level JIRA > > > Key: PARQUET-1551 > URL: https://issues.apache.org/jira/browse/PARQUET-1551 > Project: Parquet > Issue Type: Task >Reporter: Zoltan Ivanfi >Priority: Major > > This JIRA groups all other JIRA-s related to Java 11. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PARQUET-1533) TestSnappy() throws OOM exception with Parquet-1485 change
[ https://issues.apache.org/jira/browse/PARQUET-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated PARQUET-1533: -- Fix Version/s: 1.11.0 > TestSnappy() throws OOM exception with Parquet-1485 change > --- > > Key: PARQUET-1533 > URL: https://issues.apache.org/jira/browse/PARQUET-1533 > Project: Parquet > Issue Type: Bug > Components: parquet-mr >Affects Versions: 1.11.0 > Environment: Mac OS 10.14.1 >Reporter: Xinli Shang >Assignee: Gabor Szadovszky >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > > Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0 > to 128M in total. This cause the unit test TestSnappy() failed with OOM > exception. This is on my Mac laptop. > To solve the unit test failure, we can increase the size of -Xmx from 512m to > 1024m like below. However, we need to evaluate whether or not the increase of > the initial size of direct memory usage for inputBuffer and outputBuffer will > cause real Parquet application OOM or not, if that application is not with > big enough -Xmx size. > org.apache.maven.plugins > maven-surefire-plugin > ... > -Xmx1014m > ... > For details of the exception, the commit page > ([https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b]) > has the detail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PARQUET-1364) Column Indexes: Invalid row indexes for pages starting with nulls
[
https://issues.apache.org/jira/browse/PARQUET-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1364:
--
Fix Version/s: (was: 1.11.0)
> Column Indexes: Invalid row indexes for pages starting with nulls
> -
>
> Key: PARQUET-1364
> URL: https://issues.apache.org/jira/browse/PARQUET-1364
> Project: Parquet
> Issue Type: Sub-task
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>Priority: Major
> Labels: pull-request-available
>
> The current implementation for writing managing row indexes for the pages is
> not reliable. There is a logic
> [MessageColumnIO|https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L153]
> which caches null values and flush them just *before* opening a new group.
> This logic might cause starting pages with these cached nulls which are not
> correctly counted in the written rows so the rowIndexes are incorrect. It
> does not cause any issues if all the pages are read continuously put it is a
> huge problem for column index based filtering.
> The implementation described above is really complicated and would not like
> to redesign because of the mentioned issue. It is easier to simply count the
> {{0}} repetition levels as record boundaries at the column writer level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1135) upgrade thrift and protobuf dependencies
[ https://issues.apache.org/jira/browse/PARQUET-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated PARQUET-1135: -- Fix Version/s: 1.11.0 > upgrade thrift and protobuf dependencies > > > Key: PARQUET-1135 > URL: https://issues.apache.org/jira/browse/PARQUET-1135 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Reporter: Julien Le Dem >Assignee: Julien Le Dem >Priority: Major > Fix For: 1.10.0, 1.11.0 > > > thrift 0.7.0 -> 0.9.3 > protobuf 3.2 -> 3.5.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PARQUET-1667) Close InputStream after usage
[ https://issues.apache.org/jira/browse/PARQUET-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated PARQUET-1667: -- Fix Version/s: (was: 1.11.0) > Close InputStream after usage > - > > Key: PARQUET-1667 > URL: https://issues.apache.org/jira/browse/PARQUET-1667 > Project: Parquet > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > > Make sure that the streams are closed using try-with-resources -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (PARQUET-1685) Truncate the stored min and max for String statistics to reduce the footer size
[ https://issues.apache.org/jira/browse/PARQUET-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky resolved PARQUET-1685. --- Resolution: Fixed > Truncate the stored min and max for String statistics to reduce the footer > size > > > Key: PARQUET-1685 > URL: https://issues.apache.org/jira/browse/PARQUET-1685 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.10.1 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > Iceberg has a cool feature that truncates the stored min, max statistics to > minimize the metadata size. We can borrow to truncate them in Parquet also to > reduce the size of the footer, or even the page header. Here is the code in > IceBerg > [https://github.com/apache/incubator-iceberg/blob/master/api/src/main/java/org/apache/iceberg/util/UnicodeUtil.java]. > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PARQUET-1685) Truncate the stored min and max for String statistics to reduce the footer size
[ https://issues.apache.org/jira/browse/PARQUET-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated PARQUET-1685: -- Fix Version/s: (was: 1.12.0) 1.11.0 > Truncate the stored min and max for String statistics to reduce the footer > size > > > Key: PARQUET-1685 > URL: https://issues.apache.org/jira/browse/PARQUET-1685 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.10.1 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > Iceberg has a cool feature that truncates the stored min, max statistics to > minimize the metadata size. We can borrow to truncate them in Parquet also to > reduce the size of the footer, or even the page header. Here is the code in > IceBerg > [https://github.com/apache/incubator-iceberg/blob/master/api/src/main/java/org/apache/iceberg/util/UnicodeUtil.java]. > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1685) Truncate the stored min and max for String statistics to reduce the footer size
[ https://issues.apache.org/jira/browse/PARQUET-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973152#comment-16973152 ] ASF GitHub Bot commented on PARQUET-1685: - gszadovszky commented on pull request #696: PARQUET-1685: Truncate Min/Max for Statistics URL: https://github.com/apache/parquet-mr/pull/696 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Truncate the stored min and max for String statistics to reduce the footer > size > > > Key: PARQUET-1685 > URL: https://issues.apache.org/jira/browse/PARQUET-1685 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.10.1 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Iceberg has a cool feature that truncates the stored min, max statistics to > minimize the metadata size. We can borrow to truncate them in Parquet also to > reduce the size of the footer, or even the page header. Here is the code in > IceBerg > [https://github.com/apache/incubator-iceberg/blob/master/api/src/main/java/org/apache/iceberg/util/UnicodeUtil.java]. > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (PARQUET-1556) Problem with Maven repo specifications in POMs of dependencies in some development environments
[
https://issues.apache.org/jira/browse/PARQUET-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky resolved PARQUET-1556.
---
Fix Version/s: (was: 1.12.0)
Resolution: Duplicate
> Problem with Maven repo specifications in POMs of dependencies in some
> development environments
> ---
>
> Key: PARQUET-1556
> URL: https://issues.apache.org/jira/browse/PARQUET-1556
> Project: Parquet
> Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Andy Grove
>Assignee: Andy Grove
>Priority: Major
> Labels: pull-request-available
>
> Running {{mvn verify}} based on the instructions in the README results in
> this error
> {code:java}
> Could not resolve dependencies for project
> org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16{code}
> As a workaround, the local {{~/.m2/settings.xml}} file can be modified to
> include the twitter maven repo:
> {code:java}
>
> twitter
> twitter
> http://maven.twttr.com
>
> {code}
> After adding this, {{mvn verify}} works. This should not be necessary though,
> since the artifact is a transitive dependency and the POM of the direct
> dependency (elephant-bird) contains the repo specification, which works in
> most environments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1687) Update release process
[
https://issues.apache.org/jira/browse/PARQUET-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1687:
--
Fix Version/s: 1.11.0
format-2.8.0
> Update release process
> --
>
> Key: PARQUET-1687
> URL: https://issues.apache.org/jira/browse/PARQUET-1687
> Project: Parquet
> Issue Type: Task
> Components: parquet-format, parquet-mr, parquet-site
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, format-2.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Our current tagging policy in the release process requires to use the same
> tag for all the release candidates which means at RC2 we remove the tag from
> RC1 head and adds again to the RC2 head and so on. It is not a good practice.
> Hard to track RCs and rewriting git history is usually not a good idea.
> Update the related scripts and docs so we will create RC tags (e.g.
> {{apache-parquet-1.11.0-rc6}}) first and add the final release tag (e.g.
> {{apache-parquet-1.11.0}}) after the vote passes.
> It is also very confusing that we always use SNAPSHOT version of the next
> release even if the previous one is not released yet. For example we have not
> released {{1.11.0}} yet (we are after the failed vote of 1.11.0-rc6) still,
> we use the version number {{1.12.0-SNAPSHOT}}. This is very confusing.
> Update the release scripts and docs to use the same {{SNAPSHOT}} version at
> RC phase and only step to the next version number after the vote passed and
> the final release tag is added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PARQUET-1690) Integer Overflow of BinaryStatistics#isSmallerThan()
[ https://issues.apache.org/jira/browse/PARQUET-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated PARQUET-1690: -- Fix Version/s: (was: 1.11.0) Removing 1.11.0 target because this is an improvement only and not a regression since 1.10.0. > Integer Overflow of BinaryStatistics#isSmallerThan() > > > Key: PARQUET-1690 > URL: https://issues.apache.org/jira/browse/PARQUET-1690 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > > "(min.length() + max.length()) < size" didn't handle integer overflow > [https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/statistics/BinaryStatistics.java#L103] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[VOTE] Release Apache Parquet 1.11.0 RC7
Hi everyone, I propose the following RC to be released as official Apache Parquet 1.11.0 release. The commit id is 18519eb8e059865652eee3ff0e8593f126701da4 * This corresponds to the tag: apache-parquet-1.11.0-rc7 * https://github.com/apache/parquet-mr/tree/18519eb8e059865652eee3ff0e8593f126701da4 The release tarball, signature, and checksums are here: * https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc7 You can find the KEYS file here: * https://apache.org/dist/parquet/KEYS Binary artifacts are staged in Nexus here: * https://repository.apache.org/content/groups/staging/org/apache/parquet/ This release includes the changes listed at: https://github.com/apache/parquet-mr/blob/apache-parquet-1.11.0-rc7/CHANGES.md Please download, verify, and test. Please vote in the next 72 hours. [ ] +1 Release this as Apache Parquet 1.11.0 [ ] +0 [ ] -1 Do not release this because...
