[
https://issues.apache.org/jira/browse/BEAM-9188?focusedWorklogId=378651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378651
]
ASF GitHub Bot logged work on BEAM-9188:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Jan/20 06:59
Start Date: 29/Jan/20 06:59
Worklog Time Spent: 10m
Work Description: stankiewicz commented on pull request #10701:
[BEAM-9188] CassandraIO split performance improvement - cache size of the table
URL: https://github.com/apache/beam/pull/10701#discussion_r372215344
##########
File path:
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
##########
@@ -450,6 +456,10 @@ private static String buildQuery(Read spec) {
.map(ColumnMetadata::getName)
.collect(Collectors.joining(","));
+ List<TokenRange> tokenRanges =
+ getTokenRanges(cluster, spec.keyspace().get(), spec.table().get());
+ final long estimatedSize =
getEstimatedSizeBytesFromTokenRanges(tokenRanges);
Review comment:
That's good point. Based on past implementation I assumed very bad
estimation but I didn't want to change this logic.
What I could try to do is based on root estimates (SA) and n splits
calculate SA_1 = SA/n etc. Generally this size estimates is also broken from
beginning as CassandraIO can query 1 column instead of all or have filters and
this estimation doesn't take it into account. It supports only size(t) where
query is select * from table t. In A_1,A_2.. it's difficult to calculate size
as ranges are concatenated to underlying queries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 378651)
Time Spent: 2h (was: 1h 50m)
> Improving speed of splitting for Custom Sources
> -----------------------------------------------
>
> Key: BEAM-9188
> URL: https://issues.apache.org/jira/browse/BEAM-9188
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Radosław Stankiewicz
> Assignee: Radosław Stankiewicz
> Priority: Minor
> Time Spent: 2h
> Remaining Estimate: 0h
>
> At this moment Custom Source in being split and serialized in sequence. If
> there are many splits, it takes time to process all splits.
>
> Example: it takes 2s to calculate size and serialize CassandraSource due to
> connection setup and teardown. With 100+ splits, it's a lot of time spent in
> 1 worker.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
