[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91433=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91433
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 16/Apr/18 18:49
Start Date: 16/Apr/18 18:49
Worklog Time Spent: 10m
Work Description: XuMingmin closed pull request #5079: [BEAM-2990]
support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
index f32b6ce5d84..5caa6464556 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
@@ -96,9 +96,19 @@ private static long estimatedSizeBytes(FieldType
typeDescriptor, Object value) {
List list = (List) value;
long listSizeBytes = 0;
for (Object elem : list) {
- listSizeBytes +=
estimatedSizeBytes(typeDescriptor.getComponentType(), elem);
+ listSizeBytes +=
estimatedSizeBytes(typeDescriptor.getCollectionElementType(), elem);
}
return 4 + listSizeBytes;
+ case MAP:
+Map map = (Map) value;
+long mapSizeBytes = 0;
+for (Map.Entry elem : map.entrySet()) {
+ mapSizeBytes +=
typeDescriptor.getMapKeyType().equals(TypeName.STRING)
+? ((String) elem.getKey()).length()
+ : ESTIMATED_FIELD_SIZES.get(typeDescriptor.getMapKeyType());
+ mapSizeBytes += estimatedSizeBytes(typeDescriptor.getMapValueType(),
elem.getValue());
+}
+return 4 + mapSizeBytes;
case STRING:
// Not always accurate - String.getBytes().length() would be more
accurate here, but slower.
return ((String) value).length();
@@ -121,7 +131,10 @@ public Schema getSchema() {
Coder getCoder(FieldType fieldType) {
if (TypeName.ARRAY.equals(fieldType.getTypeName())) {
- return ListCoder.of(getCoder(fieldType.getComponentType()));
+ return ListCoder.of(getCoder(fieldType.getCollectionElementType()));
+} else if (TypeName.MAP.equals(fieldType.getTypeName())) {
+ return MapCoder.of(coderForPrimitiveType(fieldType.getMapKeyType()),
+ getCoder(fieldType.getMapValueType()));
} else if (TypeName.ROW.equals((fieldType.getTypeName( {
return RowCoder.of(fieldType.getRowSchema());
} else {
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index 3a7bc346883..436941b3875 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -122,8 +122,9 @@ public Builder addBooleanField(String name, boolean
nullable) {
return this;
}
-public Builder addArrayField(String name, FieldType componentType) {
- fields.add(Field.of(name,
TypeName.ARRAY.type().withComponentType(componentType)));
+public Builder addArrayField(String name, FieldType collectionElementType)
{
+ fields.add(
+ Field.of(name,
TypeName.ARRAY.type().withCollectionElementType(collectionElementType)));
return this;
}
@@ -199,6 +200,7 @@ public int hashCode() {
DATETIME, // Date and time.
BOOLEAN, // Boolean.
ARRAY,
+MAP,
ROW;// The field is itself a nested row.
private final FieldType fieldType = FieldType.of(this);
@@ -207,9 +209,13 @@ public int hashCode() {
BYTE, INT16, INT32, INT64, DECIMAL, FLOAT, DOUBLE);
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
-public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set COLLECTION_TYPES =
ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
public static final Set COMPOSITE_TYPES = ImmutableSet.of(ROW);
+public boolean isPrimitiveType() {
+ return !isCollectionType() && !isMapType() && !isCompositeType();
+}
public boolean isNumericType() {
return NUMERIC_TYPES.contains(this);
}
@@ -219,8 +225,11 @@ public boolean isStringType() {
public boolean isDateType() {
return DATE_TYPES.contains(this);
}
-public boolean isContainerType() {
- return CONTAINER_TYPES.contains(this);
+public boolean
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91401 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 16/Apr/18 17:03 Start Date: 16/Apr/18 17:03 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-381677564 Appreciate @reuvenlax , squash and merging This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91401) Time Spent: 6h (was: 5h 50m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91160 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 16/Apr/18 05:39 Start Date: 16/Apr/18 05:39 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-381485264 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91160) Time Spent: 5h 50m (was: 5h 40m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91141
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 15/Apr/18 22:47
Start Date: 15/Apr/18 22:47
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181600078
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -252,7 +264,9 @@ public FieldType type() {
@AutoValue.Builder
abstract static class Builder {
abstract Builder setTypeName(TypeName typeName);
- abstract Builder setComponentType(@Nullable FieldType componentType);
+ abstract Builder setCollectionType(@Nullable FieldType collectionType);
Review comment:
updated, please have a look when you've time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 91141)
Time Spent: 5h 40m (was: 5.5h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91140
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 15/Apr/18 22:35
Start Date: 15/Apr/18 22:35
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181599728
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -252,7 +264,9 @@ public FieldType type() {
@AutoValue.Builder
abstract static class Builder {
abstract Builder setTypeName(TypeName typeName);
- abstract Builder setComponentType(@Nullable FieldType componentType);
+ abstract Builder setCollectionType(@Nullable FieldType collectionType);
Review comment:
ok, LGTM from me once that change is made.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 91140)
Time Spent: 5.5h (was: 5h 20m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91139
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 15/Apr/18 22:23
Start Date: 15/Apr/18 22:23
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181599355
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -252,7 +264,9 @@ public FieldType type() {
@AutoValue.Builder
abstract static class Builder {
abstract Builder setTypeName(TypeName typeName);
- abstract Builder setComponentType(@Nullable FieldType componentType);
+ abstract Builder setCollectionType(@Nullable FieldType collectionType);
Review comment:
ok, let me change it to `CollectionElementType`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 91139)
Time Spent: 5h 20m (was: 5h 10m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=9=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-9
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 15/Apr/18 07:33
Start Date: 15/Apr/18 07:33
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181573132
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -252,7 +264,9 @@ public FieldType type() {
@AutoValue.Builder
abstract static class Builder {
abstract Builder setTypeName(TypeName typeName);
- abstract Builder setComponentType(@Nullable FieldType componentType);
+ abstract Builder setCollectionType(@Nullable FieldType collectionType);
Review comment:
I didn\t mind componentType, but if you want to include collection in there
then maybe CollectionElementType or CollectionValueType?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 9)
Time Spent: 5h 10m (was: 5h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91104
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 15/Apr/18 00:21
Start Date: 15/Apr/18 00:21
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181567035
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -252,7 +264,9 @@ public FieldType type() {
@AutoValue.Builder
abstract static class Builder {
abstract Builder setTypeName(TypeName typeName);
- abstract Builder setComponentType(@Nullable FieldType componentType);
+ abstract Builder setCollectionType(@Nullable FieldType collectionType);
Review comment:
Then, what's your prefer?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 91104)
Time Spent: 5h (was: 4h 50m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=91103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91103
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 14/Apr/18 23:37
Start Date: 14/Apr/18 23:37
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181566422
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -252,7 +264,9 @@ public FieldType type() {
@AutoValue.Builder
abstract static class Builder {
abstract Builder setTypeName(TypeName typeName);
- abstract Builder setComponentType(@Nullable FieldType componentType);
+ abstract Builder setCollectionType(@Nullable FieldType collectionType);
Review comment:
nitpick here: collectionType reads weird to me, as it seems like it's the
type of collection (e.g. array, list, tree, etc.) instead of the type of the
component elements of the collection.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 91103)
Time Spent: 4h 50m (was: 4h 40m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90897
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 13/Apr/18 17:10
Start Date: 13/Apr/18 17:10
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181452879
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -208,6 +209,7 @@ public int hashCode() {
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
Review comment:
In Java, container extends `Collection` and map extends `Map`, they're very
different IMO. If we merge them together I don't see any benefit as this is a
backend function and developers are using either
`TypeName.ARRAY.type().withComponentType()` or
`TypeName.MAP.type().withMapType`.
To make it clear, I would prefer to use the term **Collection** instead of
*Component* or *Contianer*. Any comments?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90897)
Time Spent: 4h 40m (was: 4.5h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90894
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 13/Apr/18 17:01
Start Date: 13/Apr/18 17:01
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181450896
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -208,8 +209,12 @@ public int hashCode() {
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
public static final Set COMPOSITE_TYPES = ImmutableSet.of(ROW);
+public boolean isPrimitiveType() {
+ return isNumericType() || isStringType() || isDateType();
Review comment:
right, will change
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90894)
Time Spent: 4.5h (was: 4h 20m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90756
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 13/Apr/18 09:45
Start Date: 13/Apr/18 09:45
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181337452
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
##
@@ -355,6 +379,11 @@ public Builder addValues(Object ... values) {
return this;
}
+public Builder addMap(Map data) {
Review comment:
> **XuMingmin** wrote:
> it's not necessary, will remove.
Acknowledged.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90756)
Time Spent: 4h 20m (was: 4h 10m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90755
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 13/Apr/18 09:45
Start Date: 13/Apr/18 09:45
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181337451
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -208,6 +209,7 @@ public int hashCode() {
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
Review comment:
> **XuMingmin** wrote:
> would separate here, CONTAINER should be ARRAY/SET. `List>` could be
a CONTAINER_TYPE, `Map<>` is not.
I'm still confused about this. In most systems, Map is considered a
container (e.g. in Java Map is a container type)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90755)
Time Spent: 4h 10m (was: 4h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90754
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 13/Apr/18 09:45
Start Date: 13/Apr/18 09:45
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r181337447
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -208,8 +209,12 @@ public int hashCode() {
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
public static final Set COMPOSITE_TYPES = ImmutableSet.of(ROW);
+public boolean isPrimitiveType() {
+ return isNumericType() || isStringType() || isDateType();
Review comment:
this is not correct (e.g. it excludes boolean). better off making this
exclusive (return !isContainterType() && !isCompositeType()).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90754)
Time Spent: 4h (was: 3h 50m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90719 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 13/Apr/18 07:03 Start Date: 13/Apr/18 07:03 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-381043689 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90719) Time Spent: 3h 50m (was: 3h 40m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90693 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 13/Apr/18 04:44 Start Date: 13/Apr/18 04:44 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-381021910 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90693) Time Spent: 3h 40m (was: 3.5h) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90692 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 13/Apr/18 04:42 Start Date: 13/Apr/18 04:42 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-381021696 any comments on the change? Would like to close this PR asap as my repository is broken after #4964 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90692) Time Spent: 3.5h (was: 3h 20m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90283 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 12/Apr/18 05:28 Start Date: 12/Apr/18 05:28 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-380682698 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90283) Time Spent: 3h 20m (was: 3h 10m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90245
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 12/Apr/18 00:59
Start Date: 12/Apr/18 00:59
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180939620
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -208,6 +209,7 @@ public int hashCode() {
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
Review comment:
would separate here, CONTAINER should be ARRAY/SET. `List>` could be a
CONTAINER_TYPE, `Map<>` is not.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90245)
Time Spent: 3h 10m (was: 3h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90244
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 12/Apr/18 00:56
Start Date: 12/Apr/18 00:56
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180939363
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -242,6 +247,10 @@ public FieldType type() {
public abstract TypeName getTypeName();
// For container types (e.g. ARRAY), returns the type of the contained
element.
@Nullable public abstract FieldType getComponentType();
+// For MAP type, returns the type of the key element.
+@Nullable public abstract FieldType getComponentKeyType();
+// For MAP type, returns the type of the value element.
+@Nullable public abstract FieldType getComponentValueType();
Review comment:
+1, will change to key as primitive, and value can be primitive/array/map/row
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90244)
Time Spent: 3h (was: 2h 50m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90242=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90242
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 12/Apr/18 00:55
Start Date: 12/Apr/18 00:55
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180939261
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
##
@@ -174,6 +176,24 @@ public void testCreatesArrayArray() {
assertEquals(data, row.getArray("array"));
}
+ @Test
+ public void testCreatesMap() {
+Map data = new HashMap() {
+ {
+put(1, "value1");
+put(2, "value2");
+put(3, "value3");
+put(4, "value4");
Review comment:
will update to support primitive/array/map/row as value type in Map
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90242)
Time Spent: 2h 50m (was: 2h 40m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90229
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 23:48
Start Date: 11/Apr/18 23:48
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180930489
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
##
@@ -355,6 +379,11 @@ public Builder addValues(Object ... values) {
return this;
}
+public Builder addMap(Map data) {
Review comment:
it's not necessary, will remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90229)
Time Spent: 2h 40m (was: 2.5h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89878
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 10:25
Start Date: 11/Apr/18 10:25
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180705567
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
##
@@ -355,6 +379,11 @@ public Builder addValues(Object ... values) {
return this;
}
+public Builder addMap(Map data) {
Review comment:
Is this overload needed? addArray was needed because otherwise passing an
array into addValues tended to unroll the array, but I don't think Java will do
that for maps.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89878)
Time Spent: 1h 50m (was: 1h 40m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89882
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 10:25
Start Date: 11/Apr/18 10:25
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180705571
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -222,6 +224,9 @@ public boolean isDateType() {
public boolean isContainerType() {
Review comment:
> **XuMingmin** wrote:
> while, my thought is MAP-> MAP, ARRAY -> CONTAINER, ROW -> COMPOSITE, to
make it clear for the backend types.
I think it's actually just another type of container. i.e. one way of
imagining .a map is it's just a container of key-value pairs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89882)
Time Spent: 2h 20m (was: 2h 10m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89883
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 10:25
Start Date: 11/Apr/18 10:25
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180705570
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -242,6 +247,10 @@ public FieldType type() {
public abstract TypeName getTypeName();
// For container types (e.g. ARRAY), returns the type of the contained
element.
@Nullable public abstract FieldType getComponentType();
+// For MAP type, returns the type of the key element.
+@Nullable public abstract FieldType getComponentKeyType();
+// For MAP type, returns the type of the value element.
+@Nullable public abstract FieldType getComponentValueType();
Review comment:
> **XuMingmin** wrote:
> I would prefer to have different fields for MAP and ARRAY, if it doesn't
cause significant performance issue.
two questions/comments
1. I think the key type be a TypeName instead of a FieldType? Making it a
FieldType makes it legal for the key to be a complex value - e.g. they key
could be an array type - which doesn't sound very meaningful to me.
2. Have you considered introducing a new key-value type here (pair of
TypeName, FieldType)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89883)
Time Spent: 2.5h (was: 2h 20m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89879
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 10:25
Start Date: 11/Apr/18 10:25
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180705565
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
##
@@ -174,6 +176,24 @@ public void testCreatesArrayArray() {
assertEquals(data, row.getArray("array"));
}
+ @Test
+ public void testCreatesMap() {
+Map data = new HashMap() {
+ {
+put(1, "value1");
+put(2, "value2");
+put(3, "value3");
+put(4, "value4");
Review comment:
I would also add a test where the map value is itself a complex or nested
type.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89879)
Time Spent: 1h 50m (was: 1h 40m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89880
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 10:25
Start Date: 11/Apr/18 10:25
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180705566
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
##
@@ -417,6 +450,22 @@ public Builder addArray(Object ... values) {
return verifiedList;
}
+private Map verifyMap(Object value, FieldType
componentKeyType,
+FieldType componentValueType, String fieldName) {
+ if (!(value instanceof Map)) {
+throw new IllegalArgumentException(String.format(
+"For field name %s and map type expected Map class. Instead " +
"class type was %s.",
+fieldName, value.getClass()));
+ }
+ Map valueMap = (Map) value;
+ Map verifiedMap =
Maps.newHashMapWithExpectedSize(valueMap.size());
+ for (Entry kv : valueMap.entrySet()) {
+verifiedMap.put(verifyPrimitiveType(kv.getKey(),
componentKeyType.getTypeName(), fieldName),
Review comment:
This is actually incorrect right now, since as coded the key type might not
be a primitive type. However as I mentioned in Schema.java, I think it would be
better to change the key type to be a TypeName, in which case this logic will
be correct.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89880)
Time Spent: 2h (was: 1h 50m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89881
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 11/Apr/18 10:25
Start Date: 11/Apr/18 10:25
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180705569
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -208,6 +209,7 @@ public int hashCode() {
public static final Set STRING_TYPES = ImmutableSet.of(STRING);
public static final Set DATE_TYPES = ImmutableSet.of(DATETIME);
public static final Set CONTAINER_TYPES = ImmutableSet.of(ARRAY);
+public static final Set MAP_TYPES = ImmutableSet.of(MAP);
Review comment:
should we consider this part of CONTAINER_TYPES?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89881)
Time Spent: 2h 10m (was: 2h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89808 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 11/Apr/18 06:50 Start Date: 11/Apr/18 06:50 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-380346045 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89808) Time Spent: 1h 40m (was: 1.5h) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89795 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 11/Apr/18 06:22 Start Date: 11/Apr/18 06:22 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-380340663 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89795) Time Spent: 1.5h (was: 1h 20m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89562=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89562
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 10/Apr/18 19:10
Start Date: 10/Apr/18 19:10
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180536885
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -222,6 +224,9 @@ public boolean isDateType() {
public boolean isContainerType() {
Review comment:
while, my thought is MAP-> MAP, ARRAY -> CONTAINER, ROW -> COMPOSITE, to
make it clear for the backend types.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89562)
Time Spent: 1h 10m (was: 1h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89560
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 10/Apr/18 19:09
Start Date: 10/Apr/18 19:09
Worklog Time Spent: 10m
Work Description: XuMingmin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180536469
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -242,6 +247,10 @@ public FieldType type() {
public abstract TypeName getTypeName();
// For container types (e.g. ARRAY), returns the type of the contained
element.
@Nullable public abstract FieldType getComponentType();
+// For MAP type, returns the type of the key element.
+@Nullable public abstract FieldType getComponentKeyType();
+// For MAP type, returns the type of the value element.
+@Nullable public abstract FieldType getComponentValueType();
Review comment:
I would prefer to have different fields for MAP and ARRAY, if it doesn't
cause significant performance issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89560)
Time Spent: 1h (was: 50m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89482
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 10/Apr/18 16:38
Start Date: 10/Apr/18 16:38
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180487493
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -242,6 +247,10 @@ public FieldType type() {
public abstract TypeName getTypeName();
// For container types (e.g. ARRAY), returns the type of the contained
element.
@Nullable public abstract FieldType getComponentType();
+// For MAP type, returns the type of the key element.
+@Nullable public abstract FieldType getComponentKeyType();
+// For MAP type, returns the type of the value element.
+@Nullable public abstract FieldType getComponentValueType();
Review comment:
to me it looks like both `getComponentValueType()` and `getComponentType()`
mean the same thing, i.e. they both describe the type of the value in the
container. Keep just one of them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89482)
Time Spent: 40m (was: 0.5h)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89483
]
ASF GitHub Bot logged work on BEAM-2990:
Author: ASF GitHub Bot
Created on: 10/Apr/18 16:38
Start Date: 10/Apr/18 16:38
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5079:
[BEAM-2990] support MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#discussion_r180488663
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
##
@@ -222,6 +224,9 @@ public boolean isDateType() {
public boolean isContainerType() {
Review comment:
Is map a container type as well? A composite type?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 89483)
Time Spent: 50m (was: 40m)
> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89274 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 10/Apr/18 06:13 Start Date: 10/Apr/18 06:13 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-379985683 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89274) Time Spent: 0.5h (was: 20m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89259 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 10/Apr/18 05:14 Start Date: 10/Apr/18 05:14 Worklog Time Spent: 10m Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079#issuecomment-379976290 R: + @akedin Can you guys take a look and let's try to finish it before 2.5 cutoff? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89259) Time Spent: 20m (was: 10m) > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-2990) support data type MAP
[ https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=89258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89258 ] ASF GitHub Bot logged work on BEAM-2990: Author: ASF GitHub Bot Created on: 10/Apr/18 05:12 Start Date: 10/Apr/18 05:12 Worklog Time Spent: 10m Work Description: XuMingmin opened a new pull request #5079: [BEAM-2990] support MAP in SQL schema URL: https://github.com/apache/beam/pull/5079 Add type MAP. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89258) Time Spent: 10m Remaining Estimate: 0h > support data type MAP > - > > Key: BEAM-2990 > URL: https://issues.apache.org/jira/browse/BEAM-2990 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Xu Mingmin >Assignee: Xu Mingmin >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > support Non-scalar types: > MAP Collection of keys mapped to values > ARRAY Ordered, contiguous collection that may contain duplicates -- This message was sent by Atlassian JIRA (v7.6.3#76005)
