[I] Issue with partitioned data [datafusion-ballista]

[via GitHub]

mshauneu opened a new issue, #1239:
URL: https://github.com/apache/datafusion-ballista/issues/1239

   **Describe the bug**
   Get an error when query a partitioned data:
   ```
   Error: ArrowError(ExternalError(Execution("Job i0BByln failed: Error 
planning job i0BByln: DataFusionError(Context(\"type_coercion\", 
SchemaError(DuplicateQualifiedField { qualifier: Bare { table: \"bhive\" }, 
name: \"year\" }, Some(\"\"))))")), None)
   ```
   DataFusion works as expected.
   
   **To Reproduce**
   ```rust
   use ballista::prelude::SessionContextExt;
   use datafusion::arrow::datatypes::DataType;
   use datafusion::datasource::file_format::parquet::ParquetFormat;
   use datafusion::datasource::listing::{
       ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
   };
   use datafusion::error::Result;
   use datafusion::prelude::SessionContext;
   use std::sync::Arc;
   
   #[tokio::main]
   async fn main() -> Result<()> {
       test(&SessionContext::new()).await?;
       test(&SessionContext::standalone().await?).await?;
       Ok(())
   }
   
   async fn test(ctx: &SessionContext) -> Result<()> {
       let session_state = ctx.state();
   
       let listing_table_url = ListingTableUrl::parse("data/bhive")?;
   
       let table_partition_cols = vec![
           ("year".to_owned(), DataType::Int64),
           ("month".to_owned(), DataType::Int64),
       ];
   
       let file_format = ParquetFormat::new()
           .with_enable_pruning(true)
           .with_skip_metadata(true);
   
       let listing_options =
           
ListingOptions::new(Arc::new(file_format)).with_table_partition_cols(table_partition_cols);
   
       let resolved_schema = listing_options
           .infer_schema(&session_state, &listing_table_url)
           .await?;
   
       let config = ListingTableConfig::new(listing_table_url)
           .with_listing_options(listing_options)
           .with_schema(resolved_schema);
   
       let listing_table = ListingTable::try_new(config)?;
   
       ctx.register_table("bhive", Arc::new(listing_table))?;
   
       ctx.sql("SELECT year, month FROM bhive GROUP BY year, month ORDER BY 
year, month")
           .await?
           .show()
           .await?;
   
       Ok(())
   }
   ````
   
   **Expected behavior**
   ```
   +------+-------+
   | year | month |
   +------+-------+
   | 2011 | 1     |
   | 2011 | 2     |
   | 2011 | 3     |
   | 2011 | 4     |
   | 2011 | 5     |
   ...
   ```
   
   **Additional context**
   ```
   tree bhive
   bhive
   ├── year=2011
   │   ├── month=1
   │   │   └── trips_0.parquet
   │   ├── month=2
   │   │   └── trips_0.parquet
   │   ├── month=3
   │   │   └── trips_0.parquet
   │   ├── month=4
   │   │   └── trips_0.parquet
   │   ├── month=5
   │   │   └── trips_0.parquet
   ...
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org