ZhangqyTJ commented on issue #2136:
URL:
https://github.com/apache/arrow-datafusion/issues/2136#issuecomment-1089890580
> > > 2. register_object_store
> >
> >
> > > Do you modify the Ballista source code and try to register
s3-objectstore ? You need to register the objectstore to runtime env in
multiple components: Ballista Client, Ballista Scheduler and Ballista Executor.
> >
> >
> > I think the correct logic is: use the code
"sessionContext.runtime_env().register_object_store("s3",Arc::new(S3FileSystem{}));"
to register the objectstore, Scheduler and Executor to obtain the serialized
s3-objectstore through the network , instead of writing the code for
registering s3-objectstore in both Scheduler and Executor. If you do that, the
parameters of the objectstore also need to be passed over the network.
@mingmwang
>
> Yes, ideally the objectstore just need to be registered once. But
currently since there is no general serialization API for the objectstore, this
is unlike JVM languages.
@mingmwang
> > > 2. register_object_store
> >
> >
> > > Do you modify the Ballista source code and try to register
s3-objectstore ? You need to register the objectstore to runtime env in
multiple components: Ballista Client, Ballista Scheduler and Ballista Executor.
> >
> >
> > I think the correct logic is: use the code
"sessionContext.runtime_env().register_object_store("s3",Arc::new(S3FileSystem{}));"
to register the objectstore, Scheduler and Executor to obtain the serialized
s3-objectstore through the network , instead of writing the code for
registering s3-objectstore in both Scheduler and Executor. If you do that, the
parameters of the objectstore also need to be passed over the network.
@mingmwang
>
> Yes, ideally the objectstore just need to be registered once. But
currently since there is no general serialization API for the objectstore, this
is unlike JVM languages.
What is the correct way? Even if the method of registering objectstore is
written in Scheduler and Executor, how should I pass parameters to Scheduler
and Executor? For example: endpoint, username, password. @mingmwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
