[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786668#comment-17786668 ] ASF GitHub Bot commented on PARQUET-1942: - Fokko commented on PR #1193: URL: https://github.com/apache/parquet-mr/pull/1193#issuecomment-1814012249 Thanks for the review @wgtmac > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786667#comment-17786667 ] ASF GitHub Bot commented on PARQUET-1942: - Fokko merged PR #1193: URL: https://github.com/apache/parquet-mr/pull/1193 > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786196#comment-17786196 ] ASF GitHub Bot commented on PARQUET-1942: - Fokko commented on code in PR #1193: URL: https://github.com/apache/parquet-mr/pull/1193#discussion_r1393757414 ## parquet-arrow/pom.xml: ## @@ -33,7 +33,7 @@ https://parquet.apache.org -0.10.0 +14.0.0 Review Comment: ```suggestion 14.0.1 ``` > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786195#comment-17786195 ] ASF GitHub Bot commented on PARQUET-1942: - Fokko commented on PR #1193: URL: https://github.com/apache/parquet-mr/pull/1193#issuecomment-1811932634 @mapleFU Thanks for pointing out. I think we can bump it to 14.0.1 right away > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786181#comment-17786181 ] ASF GitHub Bot commented on PARQUET-1942: - mapleFU commented on PR #1193: URL: https://github.com/apache/parquet-mr/pull/1193#issuecomment-1811871202 Note that CVE in arrow, should we wait for 14.0.1 ? ( https://lists.apache.org/thread/whojbb1nlyfk8zdjrkzd8ozt0l1hrz57 ) > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[
https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786166#comment-17786166
]
ASF GitHub Bot commented on PARQUET-1942:
-
wgtmac commented on code in PR #1193:
URL: https://github.com/apache/parquet-mr/pull/1193#discussion_r1393636511
##
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/Map3Levels.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.arrow.schema;
+
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.Type;
+
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+
+/**
+ * Represents a standard 3 levels Parquet map
+ */
+class Map3Levels {
+ private final GroupType map;
+ private final GroupType repeated;
+ private final Type key;
+ private final Type value;
+
+ /**
+ * Will validate the structure of the map
+ * @param map the Parquet map
+ */
+ public Map3Levels(GroupType map) {
+if (map.getLogicalTypeAnnotation() != LogicalTypeAnnotation.mapType() ||
map.getFields().size() != 1) {
+ throw new IllegalArgumentException("invalid map type: " + map);
+}
+this.map = map;
+Type repeatedField = map.getFields().get(0);
+if (repeatedField.isPrimitive() || !repeatedField.isRepetition(REPEATED)
|| repeatedField.asGroupType().getFields().size() != 2) {
+ throw new IllegalArgumentException("invalid map key: " + map);
+}
+this.repeated = repeatedField.asGroupType();
+this.key = repeated.getFields().get(0);
+this.value = repeated.getFields().get(1);
+ }
+
+ /**
+ * @return the root list element (an optional group with one child)
Review Comment:
```suggestion
* @return the root map element (an optional group with two children)
```
##
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/Map3Levels.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.arrow.schema;
+
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.Type;
+
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+
+/**
+ * Represents a standard 3 levels Parquet map
+ */
Review Comment:
```suggestion
/**
* Represents a standard 3 levels Parquet map
* - optional map
* - repeated key_value
* - required key, optional value
*/
```
##
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/Map3Levels.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785954#comment-17785954 ] ASF GitHub Bot commented on PARQUET-1942: - wgtmac commented on PR #1193: URL: https://github.com/apache/parquet-mr/pull/1193#issuecomment-1810597808 It seems that we need to update the exclusion of japicmp > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785801#comment-17785801 ] ASF GitHub Bot commented on PARQUET-1942: - Fokko closed pull request #840: PARQUET-1942: Bump Apache Arrow to 2.0.0 URL: https://github.com/apache/parquet-mr/pull/840 > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[
https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785800#comment-17785800
]
ASF GitHub Bot commented on PARQUET-1942:
-
Fokko opened a new pull request, #1193:
URL: https://github.com/apache/parquet-mr/pull/1193
Make sure you have checked _all_ steps below.
This bumps Arrow many versions:
- Some API's have changed around the field.
- Add support for converting a map and also added a test.
### Jira
- [ ] My PR addresses the following [Parquet
Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references
them in the PR title. For example, "PARQUET-1234: My Parquet PR"
- https://issues.apache.org/jira/browse/PARQUET-1942
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines. In
addition, my commits follow the guidelines from "[How to write a good git
commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain Javadoc that
explain what it does
> Bump Apache Arrow 2.0.0
> ---
>
> Key: PARQUET-1942
> URL: https://issues.apache.org/jira/browse/PARQUET-1942
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
>Affects Versions: 1.11.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785802#comment-17785802 ] ASF GitHub Bot commented on PARQUET-1942: - Fokko commented on PR #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-1809760449 Closing in favor of https://github.com/apache/parquet-mr/pull/1193 > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779287#comment-17779287 ] ASF GitHub Bot commented on PARQUET-1942: - wgtmac commented on PR #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-1778330423 > What's the status of this PR? Can we upgrade arrow as it's on a very old dependency and latest version has moved to 13.0.0 now ? > > Let me know if I can help in any way ? :) > > CC @Fokko @martin-g @gszadovszky Do you want to create a PR for this? BTW, how do you use parquet-arrow in your case? @psilos > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778994#comment-17778994 ] ASF GitHub Bot commented on PARQUET-1942: - martin-g commented on PR #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-1776790537 @psilos You could help by creating a new PR with Arrow 13 and by making sure that the CI pass! TravisCI is no more in use for testing on Linux ARM64! If the project is interested I could create a PR that uses the new ASF managed self-hosted runners at Github Actions! > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778993#comment-17778993 ] ASF GitHub Bot commented on PARQUET-1942: - psilos commented on PR #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-1776778084 What's the status of this PR? Can we upgrade arrow as it's on a very old dependency and latest version has moved to 13.0.0 now ? Let me know if I can help in any way ? :) CC @Fokko @martin-g @gszadovszky > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320986#comment-17320986 ] ASF GitHub Bot commented on PARQUET-1942: - martin-g commented on pull request #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-819497055 @Fokko Could you please rebase from latest master so we can see that TravisCI fails too ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319127#comment-17319127 ] ASF GitHub Bot commented on PARQUET-1942: - martin-g commented on pull request #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-817582630 @gszadovszky Fixed with https://github.com/apache/parquet-mr/pull/890 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319084#comment-17319084 ] ASF GitHub Bot commented on PARQUET-1942: - martin-g commented on pull request #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-817545488 @gszadovszky Checking ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317757#comment-17317757 ] ASF GitHub Bot commented on PARQUET-1942: - gszadovszky commented on pull request #840: URL: https://github.com/apache/parquet-mr/pull/840#issuecomment-816497216 @Fokko, It seems it still needs some additional work. @martin-g, do you have an idea how it is possible that Travis did not fail because of this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1942) Bump Apache Arrow 2.0.0
[ https://issues.apache.org/jira/browse/PARQUET-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242540#comment-17242540 ] Gabor Szadovszky commented on PARQUET-1942: --- [~fokko], do you want to work on it for 1.12.0? We would like to start the release process for 1.12.0 soon. > Bump Apache Arrow 2.0.0 > --- > > Key: PARQUET-1942 > URL: https://issues.apache.org/jira/browse/PARQUET-1942 > Project: Parquet > Issue Type: Improvement >Affects Versions: 1.11.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
