pratham76 opened a new issue, #13645:
URL: https://github.com/apache/gluten/issues/13645

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   
   Gluten translates Spark's per-bucket S3A options 
(`spark.hadoop.fs.s3a.bucket.<bucket>.<option>`) into Velox per-bucket keys 
(`s3.bucket.<bucket>.<option>`) in `getS3HiveConfig` 
(`cpp/velox/utils/ConfigExtractor.cc`). The parser assumes the bucket name ends 
at the first `.` after the prefix:
   
   ```cpp
   auto remaining = skey.substr(kSparkHadoopS3BucketPrefix.size());
   int dot = remaining.find(".");
   auto bucketName = remaining.substr(0, dot);
   auto suffix = remaining.substr(dot + 1);
   ```
   
   S3 bucket names may contain dots (e.g. `my.data.bucket`), and Hadoop S3A 
supports per-bucket options for them, because `S3AUtils.propagateBucketOptions` 
matches the `fs.s3a.bucket.<bucket>.` prefix using the real bucket name.
   
   For example, with:
   
   ```
   spark.hadoop.fs.s3a.bucket.my.data.bucket.endpoint=http://...
   ```
   
   Gluten parses the bucket as `my` and the option as `data.bucket.endpoint`. 
That option is not recognized, so the setting is **silently dropped**. The 
native scan then falls back to the global `fs.s3a.*` values (endpoint, 
credentials, region, IAM role, ...), while vanilla Spark correctly uses the 
per-bucket ones.
   
   A minor related issue: the `size_t` result of `find()` is stored in an `int`.
   
   
   ### Gluten version
   
   main branch
   
   ### Spark version
   
   Spark-3.5.x
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to