OlgaOvcharenko opened a new pull request #1139: URL: https://github.com/apache/systemds/pull/1139
This PR adds a new built-in mdedup for detecting duplicates in frames using matching dependencies (like Street 0.95, City 0.90 -> ZIP 1.0). @Shafaq-Siddiqi For simplicity, used Jaccard similarity, but if found out that Levenshtein or Jaro distance could also be used, should I also add them? To implement Jaccard similarity between strings the map with 2 args was added dist = map(Xi, "(x, y) -> UtilFunctions.jaccardSim(x, y)"). I also modified discoverFD built-in by setting diag to 1. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
