mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093349999
##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
public abstract class PipelineTranslator {
private static final Logger LOG =
LoggerFactory.getLogger(PipelineTranslator.class);
+ // Threshold to limit query plan complexity to avoid unnecessary planning
overhead. Currently this
+ // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways.
Until there's
+ // dedicated support for schema transforms, there's little value of allowing
more complex plans at
+ // this point.
+ private static final int PLAN_COMPLEXITY_THRESHOLD = 6;
Review Comment:
That was a decent value when testing. There's no point in having this
configurable, that way users would only worry about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
