soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1847838799
previous test i used spark parquet and had launched thrift server using
similar technique worked fine
Thrift sever
spark-submit \
--master 'local[*]' \
--conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
--conf spark.eventLog.enabled=false \
--conf
spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/sparkwarehouse \
--packages
'org.apache.spark:spark-sql_2.12:3.4.0,org.apache.spark:spark-hive_2.12:3.4.0' \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--name "Thrift JDBC/ODBC Server" \
--executor-memory 512m
this was just using spark and parquet
{{ config(
materialized='incremental',
file_format='parquet',
incremental_strategy='insert_overwrite',
) }}
SELECT
date(d) AS id,
d AS full_date,
EXTRACT (YEAR FROM d) AS YEAR,
EXTRACT (WEEK FROM d) AS year_week,
EXTRACT (DAY FROM d) AS year_day,
EXTRACT (YEAR FROM d) AS fiscal_year,
EXTRACT (QUARTER FROM d) AS fiscal_qtr,
EXTRACT (MONTH FROM d) AS MONTH,
date_format(d, 'MMMM') AS month_name,
EXTRACT (DOW FROM d) AS week_day,
date_format(d, 'EEEE') AS day_name,
(CASE WHEN date_format(d, 'EEEE') NOT IN ('Sunday', 'Saturday') THEN 0
ELSE 1 END) AS day_is_weekday
FROM (SELECT EXPLODE(months) AS d FROM (SELECT SEQUENCE
(TO_DATE('2000-01-01'), TO_DATE('2023-01-01'), INTERVAL 1 DAY) AS months))
DBT project.yml
name: 'sparkdbt'
version: '1.0.0'
config-version: 2
profile: 'sparkdbt'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
models:
sparkdbt:
core:
+enabled: true
+materialized: table
dim_date:
+materialized: table
works fine i was able to see table and query it using beeline and dbeaver as
well i am seeing issue with Hudi tables not sure what CONF i am missing here
I also have read examples provided
https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-dbt/README.md
could not get it to work for some reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
