[
https://issues.apache.org/jira/browse/BEAM-8406?focusedWorklogId=350106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350106
]
ASF GitHub Bot logged work on BEAM-8406:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Nov/19 23:40
Start Date: 26/Nov/19 23:40
Worklog Time Spent: 10m
Work Description: milantracy commented on pull request #10217:
[BEAM-8406] Add support for JSON format text tables
URL: https://github.com/apache/beam/pull/10217#discussion_r351034032
##########
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
##########
@@ -141,6 +156,51 @@ public LinesReadConverter() {}
}
}
+ /** Read-side converter for {@link TextJsonTable} with format {@code
'json'}. */
+ @VisibleForTesting
+ static class JsonToRow extends PTransform<PCollection<String>,
PCollection<Row>>
+ implements Serializable {
+
+ private ObjectMapper objectMapper;
+ private Schema schema;
+
+ @VisibleForTesting
+ public JsonToRow(ObjectMapper objectMapper, Schema schema) {
+ this.objectMapper = objectMapper;
+ this.schema = schema;
+ }
+
+ @Override
+ public PCollection<Row> expand(PCollection<String> input) {
+ return input
+ .apply(
+ ParDo.of(
+ new DoFn<String, Row>() {
+ @ProcessElement
+ public void processElement(ProcessContext context) {
Review comment:
I may plan to create the file specifically for all rows that don't match the
scheme, which is pretty similar to your design.
Do you think we really need a doc for that. Would you mind pointing me to
how a design is filed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 350106)
Time Spent: 1h (was: 50m)
> TextTable support JSON format
> -----------------------------
>
> Key: BEAM-8406
> URL: https://issues.apache.org/jira/browse/BEAM-8406
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
> Reporter: Rui Wang
> Assignee: Jing Chen
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Have a JSON table implementation similar to [1].
> [1]:
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
