thinkharderdev commented on issue #559:
URL: https://github.com/apache/arrow-ballista/issues/559#issuecomment-1352867946
> > Thanks @thinkharderdev . works with ballista-cli now. I have now built
scheduler and eexecutor as well with flag ballista-core/s3 and coonected to
instance with this ballista-cli. I am getting missing region error. I have
tried to set AWS_REGION and AWS_DEFAULT_REGION for both scheduler and
ballista-cli shells. but same error.
> > I have tested that this is finding the S3 location ok because create
command fails if I give non-existent path.
> > (base) kamal@Kamal:~/.aws$ ballista-cli --host localhost --port 50050
Ballista CLI v0.10.0 ❯ create external table test2 stored as csv location
's3://skamalj-s3/data.csv'; 0 rows in set. Query took 0.539 seconds. ❯ select *
from test2; [2022-12-14T18:25:28Z ERROR
ballista_core::execution_plans::distributed_query] Job KdKKEnv failed: Error
planning job KdKKEnv: DataFusionError(ObjectStore(Generic { store: "S3",
source: MissingRegion }))
DataFusionError(ArrowError(ExternalError(Execution("Job KdKKEnv failed: Error
planning job KdKKEnv: DataFusionError(ObjectStore(Generic { store: "S3",
source: MissingRegion }))")))) ❯
>
> It is also necessary to register S3 related configuration in env when the
`ballista-executor` starts:
>
> ```
> > export AWS_ACCESS_KEY_ID=XXXX
> > export AWS_SECRET_ACCESS_KEY=XXXX
> > export AWS_DEFAULT_REGION=XXXX
> > export AWS_ENDPOINT=https://xxxx
> ```
Yeah, both the scheduler and executor would need credentials for the S3 API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
