[
https://issues.apache.org/jira/browse/BEAM-9188?focusedWorklogId=378723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378723
]
ASF GitHub Bot logged work on BEAM-9188:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Jan/20 10:57
Start Date: 29/Jan/20 10:57
Worklog Time Spent: 10m
Work Description: stankiewicz commented on pull request #10701:
[BEAM-9188] CassandraIO split performance improvement - cache size of the table
URL: https://github.com/apache/beam/pull/10701#discussion_r372315977
##########
File path:
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
##########
@@ -450,6 +456,10 @@ private static String buildQuery(Read spec) {
.map(ColumnMetadata::getName)
.collect(Collectors.joining(","));
+ List<TokenRange> tokenRanges =
+ getTokenRanges(cluster, spec.keyspace().get(), spec.table().get());
+ final long estimatedSize =
getEstimatedSizeBytesFromTokenRanges(tokenRanges);
Review comment:
fixed this logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 378723)
Time Spent: 2h 20m (was: 2h 10m)
> Improving speed of splitting for Custom Sources
> -----------------------------------------------
>
> Key: BEAM-9188
> URL: https://issues.apache.org/jira/browse/BEAM-9188
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Radosław Stankiewicz
> Assignee: Radosław Stankiewicz
> Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> At this moment Custom Source in being split and serialized in sequence. If
> there are many splits, it takes time to process all splits.
>
> Example: it takes 2s to calculate size and serialize CassandraSource due to
> connection setup and teardown. With 100+ splits, it's a lot of time spent in
> 1 worker.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
