#general
@jiangzhx: @jiangzhx has joined the channel
@contact: @contact has joined the channel
@contact: Hey, i'm looking to setup pinot with GCS (in k8s) but i don't see how i'm supposed to add the plugin ? Is there some repository with built plugins or are they bundled by default with the docker images ? Thanks
@contact: well i see now that they are bundled directly within the docker image
@contact: found out that the plugin only need to be required using `-Dplugins.include=pinot-gcs` and that its bundled by default, however i'm trying out the `0.6.0-RC` and i got the following error: ```2020/11/06 09:07:02.367 ERROR [PluginManager] [main] Failed to load plugin [pinot-gcs] from dir [/opt/pinot/plugins/pinot-file-system/pinot-gcs] java.lang.IllegalArgumentException: object is not an instance of declaring class```
@noahprince8: I’m getting this too. ```java.lang.IllegalArgumentException: object is not an instance of declaring class at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main(LaunchDataIngestionJobCommand.java:52) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-b97460cc14b18d0ada4f4a3126c667c0d10dcb4c] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) ~[spark-core_2.12-3.0.1.jar:3.0.1] at
@contact: Do someone now if thats a known bug for the RC ?
@contact: It seems to only happen with the jdk11 (both 0.5 and 0.6-RC), i tested with jdk8 and it works fine
@g.kishore: Thanks @contact . @elon.azoulay can you please help here
@karinwolok1: Looks like Noah sweeped in and rescued the day! LOL . Thank you, @noahprince8 !
@contact: Well he said that doesnt work with his fix anyway
@noahprince8: The bit that wasn’t working was, I think, a different issue.
@noahprince8: You may want to try my fix
@noahprince8: I’m trying to run things in spark, and spark is grumpy at best.
@contact: i'm using the official docker image so i can't recompile it right now
@contact: is there a PR that fixes that ?
@elon.azoulay: Oh great! We have it working with pinot-0.5.0 but didn't try with 0.6.0-RC, also we're using jdk8 still. Nice fix @noahprince8!
@noahprince8: Not yet. I may include it in a different PR I’m working on. Haven’t done a PR to pinot yet, so may be faster if someone else just makes a PR with this small fix.
@noahprince8: I also haven’t tested that fix with Java 8. And I’m not sure I’d be the best person to verify this fix given I’m pretty inexperienced with Pinot.
@noahprince8: Oh… and now that I’m testing it with java 11 in the server starter… It doesn’t work. Yeah this is going to need a bigger fix. I think @fx19880617 is looking into parts of this issue right now.
@elon.azoulay: Maybe try deploying with java8 and see if it works? We use the gcs fs plugin with pinot 0.5.0 + java8
@contact: @elon.azoulay Do you mind if i reach out in private to ask few questions about your setup ?
@elon.azoulay: Not at all!
@fx19880617: @contact can you show me the command you are launching the ingestion job?
@contact: ```docker run --rm -ti \ --network=host \ --mount type=bind,source="$(pwd)",target=/tmp \ apachepinot/pinot:0.6.0-SNAPSHOT-ca8545b29-20201105-jdk8 LaunchDataIngestionJob \ -jobSpecFile /tmp/jobs/gcs_tracking_events_ingestion_job.yml```
@contact: as i said its working fine with jdk8
@fx19880617: ok
@contact: however the error is in the controller
@fx19880617: so jdk11 image doesnt work with this cmd?
@fx19880617: do u have controller config
@fx19880617: if you use gcs as deep store, then there are extra configs you need to put to register the pinotfS
@fx19880617:
@fx19880617: this is an example for s3
@contact: i configured it on the controller like advised: ```pinot.set.instance.id.to.hostname=true controller.data.dir=
@fx19880617: got it, so just for clarification, this works with jdk8 but not jdk11 right?
@contact: yep
@fx19880617: Got it, I will check and fix this in java 11
@contact: thanks a lot !
@noahprince8: @contact org/apache/pinot/spi/plugin/PluginClassLoader.java Change line 50 to: ``` method.invoke(this, url);``` instead of ``` method.invoke(classLoader, url);``` Fixed the issue for me.
@noahprince8: Though I’m still getting class not found issues, so unsure if that’s something else or this didn’t _really_ fix the problem
@jsorenson: @jsorenson has joined the channel
#random
@jiangzhx: @jiangzhx has joined the channel
@contact: @contact has joined the channel
@jsorenson: @jsorenson has joined the channel
#community
@jiangzhx: @jiangzhx has joined the channel
#time-based-segment-pruner
@noahprince8: How goes the time based segment pruner?
#segment-cold-storage
@noahprince8: Neat, got the lazy loading working. Going to add some metrics and clean this up a bit.
@noahprince8: Ah, also need to make it download direct from s3 instead of the controller. That's gonna put load on the controller for a big query
@g.kishore: That’s already supported
@g.kishore: It depends on how segment is pushed
@g.kishore: If you use uri based push, it automatically downloads it from S3
@fx19880617: @noahprince8 you can try to use `jobType: SegmentCreationAndMetadataPush` in the spec.yaml file
@fx19880617: if your output directory is already s3 ```# outputDirURI: Root directory of output segments, expected to have scheme configured in PinotFS. outputDirURI: 's3://<your-bucket>/pinot/<table>/segments'```
@noahprince8: I think server download from s3 is only done for real time tables.
@fx19880617: it’s both :slightly_smiling_face:
@fx19880617: you also need to configure pinot fs on server as well
@fx19880617:
@fx19880617: this tutorial uses segment uri push, since you already on newest version, segment metadata push will be much better
@noahprince8: Is there documentation on what the difference between these is?
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
