Beyondeclipse commented on a change in pull request #11345:
URL: https://github.com/apache/shardingsphere/pull/11345#discussion_r674815965
##########
File path:
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/builder/SchemaBuilder.java
##########
@@ -129,16 +120,40 @@ private static void appendRemainTables(final
SchemaBuilderMaterials materials, f
}
return Optional.empty();
}
-
- private static void appendDialectRemainTables(final
DialectTableMetaDataLoader dialectLoader, final SchemaBuilderMaterials
materials, final Map<String, TableMetaData> tables) throws SQLException {
+
+ private static Map<String, String> getActualTable2LogicTableMap(final
SchemaBuilderMaterials materials) {
+ List<DataNodeContainedRule> dataNodeContainedRuleList =
materials.getRules().stream()
+ .filter(each -> each instanceof DataNodeContainedRule)
+ .map(each -> (DataNodeContainedRule) each)
+ .collect(Collectors.toList());
+ if (CollectionUtils.isEmpty(dataNodeContainedRuleList)) {
+ return Collections.emptyMap();
+ }
+
+ int size = dataNodeContainedRuleList.stream()
+ .mapToInt(each ->
each.getAllDataNodes().entrySet().stream().mapToInt(entry ->
entry.getValue().size()).sum())
+ .sum();
+ Map<String, String> actualTable2LogicTableMap = new HashMap<>(size);
+ for (DataNodeContainedRule each : dataNodeContainedRuleList) {
+ for (Entry<String, Collection<DataNode>> entry :
each.getAllDataNodes().entrySet()) {
+ List<String> actualTables =
entry.getValue().stream().map(DataNode::getTableName).collect(Collectors.toList());
+ for (String oneActualTalbe : actualTables) {
+ actualTable2LogicTableMap.put(oneActualTalbe,
entry.getKey());
+ }
+ }
+ }
+ return actualTable2LogicTableMap;
+ }
+
+ private static void appendDialectRemainTables(final
DialectTableMetaDataLoader dialectLoader, final SchemaBuilderMaterials
materials,
+ final Map<String, String> ruleActualTable2LogicTableMap, final
Map<String, TableMetaData> tables) throws SQLException {
Collection<Future<Map<String, TableMetaData>>> futures = new
LinkedList<>();
- Collection<String> existedTables =
getExistedTables(materials.getRules(), tables);
for (DataSource each : materials.getDataSourceMap().values()) {
- futures.add(EXECUTOR_SERVICE.submit(() -> dialectLoader.load(each,
existedTables)));
+ futures.add(EXECUTOR_SERVICE.submit(() ->
dialectLoader.load(each)));
Review comment:
@strongduanmu The original code also loaded all the tables, but use
single-thread instand. And in my environment, this change reduce the loading
time from 5m to 50s with 400+ databases each has 50+ tables.
In fact, the performance issue of original code is opening many times
connections for the same database. And this change is try to improve it by
reducing connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
