tustvold commented on code in PR #260:
URL: https://github.com/apache/arrow-ballista/pull/260#discussion_r978405703
##########
ballista/rust/core/src/utils.rs:
##########
@@ -54,9 +58,47 @@ use std::time::Duration;
use std::{fs::File, pin::Pin};
use tonic::codegen::StdError;
use tonic::transport::{Channel, Error, Server};
+use url::Url;
-/// Stream data to disk in Arrow IPC format
+/// Default session builder using the provided configuration
+pub fn default_session_builder(config: SessionConfig) -> SessionState {
+ SessionState::with_config_rt(
+ config,
+ Arc::new(
+
RuntimeEnv::new(with_object_store_provider(RuntimeConfig::default()))
+ .unwrap(),
+ ),
+ )
+}
+/// Get a RuntimeConfig with specific ObjectStoreDetector in the
ObjectStoreRegistry
+pub fn with_object_store_provider(config: RuntimeConfig) -> RuntimeConfig {
+
config.with_object_store_registry(Arc::new(ObjectStoreRegistry::new_with_provider(
+ Some(Arc::new(FeatureBasedObjectStoreProvider)),
+ )))
+}
+
+/// An object store detector based on which features are enable for different
kinds of object stores
+pub struct FeatureBasedObjectStoreProvider;
+
+impl ObjectStoreProvider for FeatureBasedObjectStoreProvider {
+ /// Detector a suitable object store based on its url if possible
+ /// Return the key and object store
+ #[allow(unused_variables)]
+ fn get_by_url(&self, url: &Url) -> Option<Arc<dyn ObjectStore>> {
+ #[cfg(feature = "hdfs")]
+ {
+ let store = HadoopFileSystem::new(url.as_str());
+ if let Some(store) = store {
+ return Some(Arc::new(store));
+ }
+ }
+
+ None
Review Comment:
Following the current construction this could return
"FeatureBasedObjectStoreProvider failed to create store for url {}"
This is already an improvement as it tells us that the provider was
registered and used.
As described above, I think it would be even better if it could provide
insight into why nothing was found, e.g. not compiled with the right feature,
unsupported scheme, etc..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
