yifan-c commented on code in PR #239:
URL:
https://github.com/apache/cassandra-analytics/pull/239#discussion_r3921500375
##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/bulkwriter/SortedSSTableWriter.java:
##########
@@ -289,19 +295,31 @@ public synchronized void close(BulkWriterContext
writerContext) throws IOExcepti
// FIXME: CQLSSTableWriter may produce incomplete Filter.db file,
rebuilding it manually (see CASSANDRA-21423).
rebuildFilterComponents(writerContext, unhashedFilter);
- try (DirectoryStream<Path> dataFileStream =
getDataFileStream(unhashedFilter))
+ PreparedSSTables prepared = new PreparedSSTables();
+
+ try (DirectoryStream<Path> fileStream =
Files.newDirectoryStream(getOutDir(), unhashedFilter))
{
- for (Path dataFile : dataFileStream)
+ for (Path path : fileStream)
{
// NOTE: We calculate file hashes before re-reading so that we
know what we hashed
// is what we validated. Then we send these along with
the files and the
// receiving end re-hashes the files to make sure they
still match.
- Map<Path, Digest> newFileDigests =
calculateFileDigestMap(dataFile);
- overallFileDigests.putAll(newFileDigests);
- newlyHashedFiles.addAll(newFileDigests.keySet());
- sstableCount += 1;
+ Digest digest = digestAlgorithm.calculateFileDigest(path);
+ LOGGER.debug("Calculated digest={} for path={}", digest, path);
+
+ overallFileDigests.put(path, digest);
+ newlyHashedFiles.add(path);
+
+ prepared.getOrPrepareSSTable(path)
+ .addComponent(path, digest);
+
+ if (isDataFile(path))
+ {
+ sstableCount += 1;
+ }
Review Comment:
nit: how about adding the check outside of the for-loop to assert that all
`PreparedSSTable` have non-null data file in `prepared`.
Then, the if statement can be replaced with `sstableCount +=
prepared.sstables().size();`.
The same applies to `prepareSStablesToSend`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
