<h3><u>#general</u></h3><br><strong>@vickykak: </strong>@vickykak has joined
the channel<br><strong>@rgarcia: </strong>@rgarcia has joined the
channel<br><strong>@brandenc: </strong>@brandenc has joined the
channel<br><h3><u>#random</u></h3><br><strong>@vickykak: </strong>@vickykak has
joined the channel<br><strong>@alan.hardy: </strong>@alan.hardy has left the
channel<br><strong>@rgarcia: </strong>@rgarcia has joined the
channel<br><strong>@brandenc: </strong>@brandenc has joined the
channel<br><h3><u>#troubleshooting</u></h3><br><strong>@somanshu.jindal:
</strong>@npawar I have attached new snapshot of error along with zooinspector
snapshot.<br><strong>@somanshu.jindal: </strong>I think commit is failing for
me from the start. Generally i had longer commit time or rows so i never
checked. I got this issue when i was verifying commit by setting lower values
in table config.<br><strong>@alan.hardy: </strong>Hi. I'm standing up a
Kubernetes/Helm cluster as per docs. Modified to use S3, again as per docs:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMdTeAXadp8BL3QinSdRtJdrQSnuJvsa-2BKLAdxKaLYoSk5fnxow1nUtu2wwx55WLnNxUjAtupORoTsl-2B1z4-2FRqPw-3Dys-3_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0Jh1O7ykR84JxmF9yzYD4CwbIHHutAQG6GpnDvObyruJS4x9-2FKDyF7ezgPxFQ7jH39xSlcQloPtSHs1cYzLHfCnrmXn51lv1BzGjsN0oBbjly2RgZDQ-2B-2B-2BoP-2BekLyV10OA-2FSCwSHHbEj6vVw-2Fhkc39l8CzHvyCYOKVrvj4S0PKxc-3D>.
Controller & Server are throwing
```ERROR [PinotFSFactory] [main] Could not instantiate file system for class
org.apache.pinot.plugin.filesystem.S3PinotFS with scheme s3
java.lang.ClassNotFoundException: org.apache.pinot.plugin.filesystem.S3PinotFS
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_252]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_252]
at
org.apache.pinot.spi.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:80)
~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]```
<br><strong>@pradeepgv42: </strong>Hi, I am trying to test following change
(<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKwnNfTsLQ-2Fmo2mEKJSpKxog-3D-3DacUg_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0dE4b0jBVzO87Th5SrhtOxGqa-2BXHuoohApXHM9K9CucP2Rxg-2B-2Fbrq7ptiXoD9GEjgz5fmPJAi-2BeVKsUEj2lkEp49q-2B5t-2Bg9Wh3YogiBuAX7JspqyNzXs71ywSDIOIIX9Jbjv-2FRWFTWC-2BI0ShI0w7qX-2B-2FH32r6Qe7GKnOzUH-2B1xsw-3D>)
on my cluster. So, pulled code from the master, but I am seeing below
exception. wondering if there’s any change you know of? I only see this change
(<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKvd1xlGru5RyM-2Fuil934eiw-3D-3D4UiU_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0xC-2Fk-2B74-2BxM2tPiPqq7Hmz91Ma6EralN7-2BG6So5vCioP-2Bv6Ll-2BBLXAcokwu-2FN2YctK4m-2BjfJvxP-2BZOXC2YRD0BbcmFAEQ2wTM2PGpcr-2BCIBsDhH1rTXyPr7sVLr3AwUxfrNrE1jNRbYL2SlrP62P2nXw-2Fkad8AzXj0Wm9f2KDVdc-3D>)
which says that existing behavior shouldn’t change?
Below is the exception I see, which seems to be trying to fetch the S3 region
from configuration
```
java.lang.IllegalArgumentException: null
at
shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependenci
es.jar:0.5.0-SNAPSHOT-2ec7dee1597021742f68f0ae8b279f7560e55894]
at org.apache.pinot.plugin.filesystem.S3PinotFS.init(S3PinotFS.java:80)
~[pinot-s3-0.5.0-SNAPSHOT-shaded.jar:0.5.0-SNAPSHOT-2ec7dee
1597021742f68f0ae8b279f7560e55894]
at
org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:55)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.
jar:0.5.0-SNAPSHOT-2ec7dee1597021742f68f0ae8b279f7560e55894]
at
org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:75)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:
0.5.0-SNAPSHOT-2ec7dee1597021742f68f0ae8b279f7560e55894]```<br><strong>@mayanks:
</strong>As per the code the precondition is `
```Preconditions.checkArgument(!isNullOrEmpty(config.getProperty(REGION)));```
<br><strong>@pradeepgv42:
</strong>```pinot.server.storage.factory.s3.region=us-east-2```
I already have this config,<br><strong>@pradeepgv42: </strong>and this was
working fine with earlier version<br><strong>@g.kishore: </strong>@dlavoie
^^<br><strong>@dlavoie: </strong>@dlavoie has joined the
channel<br><strong>@mayanks: </strong>Yeah, #5608 seems one PR that is
related.<br><strong>@dlavoie: </strong>Definately sounds related, I’ll
investigate to tomorrow morning!<br><strong>@mayanks: </strong>My guess is
subsetting of config is broken.<br><strong>@mayanks: </strong>`
```PinotConfiguration schemesConfiguration = fsConfig.subset(CLASS);```
<br><strong>@dlavoie: </strong>Yeah, that would explain the config being object
being null.<br><strong>@mayanks: </strong>Is `class` new?<br><strong>@mayanks:
</strong>If so, this is a backward incompatible change?<br><strong>@dlavoie:
</strong>We have tests around fsConfit subsetting, if that is broken, it’s
definitely not intended. I’m not home right now.<br><strong>@g.kishore:
</strong>@pradeepgv42 can you paste the configuration<br><strong>@g.kishore:
</strong>entire file<br><strong>@pradeepgv42:
</strong>```pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.accessKey=
pinot.server.storage.factory.s3.secretKey=
pinot.server.storage.factory.s3.region=
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.dataDir=/home/ubuntu/pinot/data
pinot.server.instance.segmentTarDir=/home/ubuntu/pinot/segments```
<br><strong>@pradeepgv42: </strong>This is the server
config<br><strong>@g.kishore: </strong>tx<br><strong>@g.kishore: </strong>found
the issue:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XK1GA-2FG5rC0yUXuYJG94BOQg-3D-3DTFgm_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0F9V3n9S3OOFZn-2BYLOdN2Fcxtkj2vGB7nZN0Ifivc9-2BlVy49U1OdWPOJvE6L2CYeIrrE5clFe26DuU9urZK7bv79ECYKKU-2BvI8hBnt82Fy1BCbDDoyetXDJn75z9IcsVxoal6COgrYYamUmdwoAeZXQVSKV1bwFYmTSg4P-2FG3MiI-3D><br><strong>@pradeepgv42:
</strong>tested the fix, it’s working now thanks
@g.kishore<br><strong>@mayanks: </strong>@g.kishore Checked the callers of
`init`, going by variable names, some are passing `config` and others are
passing `fsConfig`<br><strong>@dlavoie: </strong>I'm back home, I'll review the
findings and provide more context if needed.<br><strong>@mayanks:
</strong>Thanks @dlavoie. One thing this failure brings up is that we have lack
of test coverage here. Perhaps we should also use this opportunity to improve
on that<br><h3><u>#pinot-dev</u></h3><br><strong>@ssubrama: </strong>@dlavoie
we had to do this to get a nightly build going. @jlli has more information.
Also, why do we need to release a snapshot of pinot for other projects? If you
are doing manual builds, can you provide some values for these variables and
get a snapshot build<br><strong>@dlavoie: </strong>Having a SNAPSHOT release
channel allows a CI pipeline to quickly identify potential breaking changes and
new features before the release is published. Most OSS project I have seen have
snapshot releases channel.
What I've seen work well is to always have maven version explicit in the poms
and manage version upgrade with the mvn release plugin as part of the release
process. When a release is cut, snapshot is moved to a final version, tag is
done on git, build is published to maven, then main branch is updated with the
new version snapshot.<br><strong>@dlavoie: </strong>I feel that after a git
clone, `mvn package` should just work without specific
instructions.<br><strong>@dlavoie: </strong>Also, when working locally, sub
modules of Pinot can't be built individually with `-rf :sub-module` because of
that. The whole project needs to be rebuilt.<br><strong>@jlli:
</strong>@dlavoie you don’t have to specify anything for revision and sha1 if
you want to make a snapshot build; `sha1` has its default value, which is
`-SNAPSHOT`. Plus, there is an alternative to make a release instead of using
release plugin:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMZLbmandTFEO1t1g5FzNidEeemdSSybunX9ukF8jlz3roygTevMb76DabYOpwo-2BQaNsZ2WLQVUh4-2FnC-2BcMZ40169EO3-2FjGfvrtKip-2BnK7uMp7BUq_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0Uzgn6tFD6X-2F4klGQc5EOsE1I7q-2FQvAKfS7qOx6xwpNGRPrnrAC6BPaHc0M1oJJPUoBJsjVIcyBg-2FadPBzXqv3uaa54UUjQhNpCkrgUx8iF7b87LAghfcryKB-2BWuA-2F1nJZccbMong3XVOWC2eQL5lZTfnGlr-2FeuE9VbJWEVRRXNo-3D><br><strong>@dlavoie:
</strong>Hi @jlli default values will only apply if you build from the parent
pom. I'll go through that interesting reading. Thank
you!<br><h3><u>#presto-pinot-streaming</u></h3><br><strong>@jackie.jxt:
</strong>@elon.azoulay The doc looks good.<br><strong>@jackie.jxt: </strong>Can
you please address the comments in the protobuf PR? Then we can merge it and
start the next step<br><strong>@elon.azoulay:
</strong>Sure!<br><strong>@mayanks: </strong>@elon.azoulay posted some minor
questions on the doc.<br><strong>@elon.azoulay: </strong>I'll list the future
enhancements, thanks for the
feedback!<br><h3><u>#s3-multiple-buckets</u></h3><br><strong>@mayanks:
</strong>@mayanks has joined the channel<br>
