[
https://issues.apache.org/jira/browse/FLINK-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210343#comment-15210343
]
Fabian Hueske commented on FLINK-3664:
--------------------------------------
Hi [~tlisonbee], I like this proposal a lot. It would be a great feature to add.
A few comments on you proposal:
- If possible use a {{CombineFunction}} instead of a {{GroupCombineFunction}}.
{{CombineFunction}} can be executed using a hash-based execution strategy
whereas {{GroupCombineFunction}} must be evaluated with a sort-based strategy.
Hash-based combiners are not supported yet but will hopefully be added soon
(see PR #1517).
- We add new functionality such as {{summarize()}} to {{DataSetUtils}} and not
directly to {{DataSet}}. {{DataSetUtils}} serves kind of as a staging area for
new features where we are not sure yet whether we want to add them to the core
API.
- It is certainly possible to implement the feature as {{CustomUnaryOperation}}
but it is also fine to directly call the API methods.
- Do you think it makes sense to include (approximate) distinct counts for
integer and String values or should this as a separate method?
> Create a method to easily Summarize a DataSet
> ---------------------------------------------
>
> Key: FLINK-3664
> URL: https://issues.apache.org/jira/browse/FLINK-3664
> Project: Flink
> Issue Type: Improvement
> Reporter: Todd Lisonbee
> Attachments: DataSet-Summary-Design-March2016-v1.txt
>
>
> Here is an example:
> {code}
> /**
> * Summarize a DataSet of Tuples by collecting single pass statistics for all
> columns
> */
> public Tuple summarize()
> Dataset<Tuple3<Double, String, Boolean>> input = // [...]
> Tuple3<DoubleColumnSummary,StringColumnSummary,BooleanColumnSummary> summary
> = input.summarize()
> summary.getField(0).stddev()
> summary.getField(1).maxStringLength()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
