GitHub user ParyshevSergey created a discussion: [GLUTEN][FLINK] Nexmark q0 performance
Hello community, according [flink docs](https://github.com/apache/incubator-gluten/blob/main/gluten-flink/docs/Flink.md) I builded velox4j myself with gluten and run local cluster to test nexmark q0 query. With the latest branch of nexmark datagen not working for me (pure virtual call issue in logs) then I try to use kafka connector with local kafka. And in this case q0 pass after 49min - I noticed that conversion between RowData and RowVector is heavy operation so I removed code in CommonExecSink of StreamRecordTimestampInserter to avoid conversion. So after run the job was (NativeKafka -> gluten_calc -> Writer) as one chained operator - result is 8min. Without gluten same config the job pass after 4min. Where is my problem? @shuai-xu @KevinyhZou What do you think? <details><summary>flink config.yaml</summary> <p> ``` taskmanager.memory.process.size: 4G jobmanager.rpc.address: localhost jobmanager.rpc.port: 6123 jobmanager.memory.process.size: 4G taskmanager.numberOfTaskSlots: 1 parallelism.default: 1 io.tmp.dirs: /tmp/flink/tmp/ jobmanager.bind-host: 0.0.0.0 rest.bind-host: 0.0.0.0 #============================================================================== # JVM #============================================================================== # JVM options for GC env.java.opts: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:NewRatio=3 -XX:ConcGCThreads=4 --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED env.java.opts.jobmanager: -Xloggc:$FLINK_LOG_DIR/jobmanager-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED env.java.opts.taskmanager: -Xloggc:$FLINK_LOG_DIR/taskmanager-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED #============================================================================== # State & Checkpoint #============================================================================== state.backend.type: rocksdb # for example, hdfs://benchmark01/checkpoint execution.checkpointing.dir: file:///tmp/flink/checkpoint execution.checkpointing.incremental: true execution.checkpointing.interval: 18000000 execution.checkpointing.mode: EXACTLY_ONCE state.backend.local-recovery: true #============================================================================== # Runtime Others #============================================================================== # configuration options for adjusting and tuning table programs. #table.exec.mini-batch.enabled: true #table.exec.mini-batch.allow-latency: 2s #table.exec.mini-batch.size: 50000 table.optimizer.distinct-agg.split.enabled: true # disable final checkpoint to avoid test waiting for the last checkpoint complete execution.checkpointing.checkpoints-after-tasks-finish.enabled: false execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION jobstore.expiration-time: 1000000 execution.checkpointing.tolerable-failed-checkpoints: 10 ``` </p> </details> GitHub link: https://github.com/apache/incubator-gluten/discussions/11508 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
