LuciferYang commented on code in PR #37843:
URL: https://github.com/apache/spark/pull/37843#discussion_r967845225
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Expression.java:
##########
@@ -44,7 +44,16 @@ public interface Expression {
* List of fields or columns that are referenced by this expression.
*/
default NamedReference[] references() {
- return Arrays.stream(children()).map(e -> e.references())
- .flatMap(Arrays::stream).distinct().toArray(NamedReference[]::new);
+ List<NamedReference> list = new ArrayList<>();
+ Set<NamedReference> uniqueValues = new HashSet<>();
+ for (Expression e : children()) {
+ NamedReference[] references = e.references();
+ for (NamedReference reference : references) {
+ if (uniqueValues.add(reference)) {
+ list.add(reference);
+ }
+ }
+ }
+ return list.toArray(new NamedReference[0]);
Review Comment:
@srowen
[8ff3b77](https://github.com/apache/spark/pull/37843/commits/8ff3b77bcebc1c2dd1bfd26a3391d950790071e2)
revert to the previous version, because from the local test, using
LinkedHashSet is slower than using ArrayList + HashSet, which may be because
LinkedHashSet.toarray is much slower than ArrayList.toarray.
Let's wait for the bench results from GA to verify multiple Java versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
