[
https://issues.apache.org/jira/browse/FLINK-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218418#comment-15218418
]
Todd Lisonbee commented on FLINK-3664:
--------------------------------------
I've completed a first pass for this implementation and would like any early
feedback,
https://github.com/tlisonbee/flink/commit/2a7ad55d704bd3188ea8ae4cbfb7f40319474eef
(the important changes you might want to look at are in Aggregator,
NumericSummaryAggregator, and DataSetUtils)
My "to do" list before submitting pull request:
- Blanket the code with comments, unit tests, and integration tests
- Incorporate any early feedback
Tasks I was planning on doing under a follow-on JIRA (not part of initial pull
request):
- Add support for more data types (unless any others seem like must-have, I can
do now)
- Add a summarize() method for GroupedDataSets
Thanks.
> Create a method to easily Summarize a DataSet
> ---------------------------------------------
>
> Key: FLINK-3664
> URL: https://issues.apache.org/jira/browse/FLINK-3664
> Project: Flink
> Issue Type: Improvement
> Reporter: Todd Lisonbee
> Attachments: DataSet-Summary-Design-March2016-v1.txt
>
>
> Here is an example:
> {code}
> /**
> * Summarize a DataSet of Tuples by collecting single pass statistics for all
> columns
> */
> public Tuple summarize()
> Dataset<Tuple3<Double, String, Boolean>> input = // [...]
> Tuple3<DoubleColumnSummary,StringColumnSummary,BooleanColumnSummary> summary
> = input.summarize()
> summary.getField(0).stddev()
> summary.getField(1).maxStringLength()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
