sekikn opened a new issue, #19174: URL: https://github.com/apache/pulsar/issues/19174
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version OS: Ubuntu 22.04 LTS Pulsar: master branch (commit id: 9ef54fd012e2f80f4499263f6ac4542b45c5bc15) Alluxio: 2.7.3 ### Minimal reproduce step Build Pulsar on the master branch and run it with the Alluxio sink: ``` $ git show --shortstat commit 9ef54fd012e2f80f4499263f6ac4542b45c5bc15 (HEAD -> master, upstream/master, origin/master, origin/HEAD) Author: AloysZhang <[email protected]> Date: Tue Jan 10 15:07:29 2023 +0800 [improve][cli] Expose updateLocalTopicOnly to UpdatePartitionedCmd (#19162) 1 file changed, 5 insertions(+), 1 deletion(-) $ ./mvnw clean install -DskipTests ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 08:54 min [INFO] Finished at: 2023-01-10T23:09:54+09:00 [INFO] ------------------------------------------------------------------------ $ mkdir connectors $ cp pulsar-io/alluxio/target/pulsar-io-alluxio-2.12.0-SNAPSHOT.nar connectors $ bin/pulsar standalone ``` Download and setup Alluxio: ``` $ curl -sLO https://downloads.alluxio.io/downloads/files/2.7.3/alluxio-2.7.3-bin.tar.gz $ tar xf alluxio-2.7.3-bin.tar.gz $ cd alluxio-2.7.3 $ bin/alluxio-start.sh local SudoMount ... ----------------------------------------- Starting to monitor all local services. ----------------------------------------- --- [ OK ] The master service @ sekikn-ThinkCentre-M75q-Gen-2 is in a healthy state. --- [ OK ] The job_master service @ sekikn-ThinkCentre-M75q-Gen-2 is in a healthy state. --- [ OK ] The worker service @ sekikn-ThinkCentre-M75q-Gen-2 is in a healthy state. --- [ OK ] The job_worker service @ sekikn-ThinkCentre-M75q-Gen-2 is in a healthy state. --- [ OK ] The proxy service @ sekikn-ThinkCentre-M75q-Gen-2 is in a healthy state. ``` ``` $ bin/alluxio fs mkdir /pulsar Successfully created directory /pulsar $ bin/alluxio fs chmod 777 /pulsar Changed permission of /pulsar to 777 ``` Register the Alluxio sink and ingest more messages than the value of `rotationRecords`. In the example below, 11 messages are ingested while `rotationRecords` is set to 10. ``` $ bin/pulsar-admin sink available-sinks alluxio Writes data into Alluxio ---------------------------------------- $ cat ~/alluxio-sink.yml configs: alluxioMasterHost: "localhost" alluxioMasterPort: "19998" alluxioDir: "pulsar" filePrefix: "TopicA" fileExtension: ".txt" lineSeparator: "\n" rotationRecords: 10 rotationInterval: "-1" $ bin/pulsar-admin sinks create --tenant public --namespace default --name alluxio-sink --sink-type alluxio --sink-config-file ~/alluxio-sink.yml --inputs TopicA Created successfully $ for i in $(seq 0 10); do bin/pulsar-client produce -m "key-$i" -n 1 TopicA; done ``` ### What did you expect to see? A new temporary file is created on Alluxio. ### What did you see instead? File creation fails with the following error. Pulsar: ``` $ cat logs/functions/public/default/alluxio-sink/alluxio-sink-0.log ... 2023-01-10T23:15:41,468+0900 [public/default/alluxio-sink-0] ERROR org.apache.pulsar.io.alluxio.sink.AlluxioSink - Unable to write record to alluxio. alluxio.exception.FileDoesNotExistException: Path "/pulsar/tmp/c436b01b-2c4c-44b6-9f76-cfa436c55133_tmp.txt" does not exist. at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:575) ~[alluxio-core-client-fs-2.7.3.jar:?] at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:164) ~[alluxio-core-client-fs-2.7.3.jar:?] at org.apache.pulsar.io.alluxio.sink.AlluxioSink.createTmpFile(AlluxioSink.java:222) ~[C7-XLukrb504fkTxC3363g/:?] at org.apache.pulsar.io.alluxio.sink.AlluxioSink.writeToAlluxio(AlluxioSink.java:198) ~[C7-XLukrb504fkTxC3363g/:?] at org.apache.pulsar.io.alluxio.sink.AlluxioSink.write(AlluxioSink.java:137) ~[C7-XLukrb504fkTxC3363g/:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.sendOutputMessage(JavaInstanceRunnable.java:429) ~[?:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.handleResult(JavaInstanceRunnable.java:391) ~[?:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:331) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] ``` Alluxio: ``` $ cat logs/master.log ... 2023-01-10 23:16:41,540 WARN FileSystemMasterClientServiceHandler - Exit (Error): CreateFile: request=path: "/pulsar/tmp/8cbf15f1-35ee-4240-992e-c77dd4befe7c_tmp.txt" options { blockSizeBytes: 67108864 recursive: false mode { ownerBits: READ_WRITE groupBits: READ otherBits: READ } replicationMax: -1 replicationMin: 0 replicationDurable: 1 writeTier: 0 writeType: MUST_CACHE commonOptions { syncIntervalMs: -1 ttl: -1 ttlAction: DELETE operationId { mostSignificantBits: 269283063510748789 leastSignificantBits: -7374140582089861256 } } persistenceWaitTime: 0 } , Error=alluxio.exception.FileDoesNotExistException: Path "/pulsar/tmp/8cbf15f1-35ee-4240-992e-c77dd4befe7c_tmp.txt" does not exist. ``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
