[ https://issues.apache.org/jira/browse/SPARK-34751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303001#comment-17303001 ]
Nivas Umapathy edited comment on SPARK-34751 at 3/17/21, 12:52 AM: ------------------------------------------------------------------- the schema is extracted from the same file, before materializing the data {{{{df = glue_context.read.schema(df.schema).parquet(}}{{'invalid_columns_double.parquet')}}}} {{{{ ^^^^^^^^^^^^^^^^^}}}} by schema I meant this. The file was written out from pandas dataframe. It was written out using this {{}} {{import pandas as pd}} {{df = pd.DataFrame(}} {{data = {}} {{"COL 1": [87.0921538,41.26487033,1.731626487,99.02779887,80.750347,62.89799664,84.27772144,12.78995399,58.13994625,54.51677768],}} {{"COL,2": [28.431596,13.50141322,28.60878912,50.59429628,53.77345338,3.278319754,89.88524435,57.29173215,34.75955608,22.50907852],}} {{"COL;3": [48.12359525,2.751809433,64.45305108,40.97279762,46.3506431,68.57561523,67.52866381,18.70752371,44.86086801,8.42884315],}} {{"COL{4": [25.23141131,65.20640894,56.83503264,21.46097087,59.22963758,99.55784318,8.02616508,75.29924438,3.911268106,90.1820556],}} {{"COL}5": [37.00662369,82.24478025,27.89576774,9.549598639,46.92239754,10.48954042,81.71312268,49.991685,43.78556399,79.00133828],}} {{"COL(6": [71.21354798,82.33860851,12.88393027,23.47301417,76.36836392,18.43024893,51.48770487,93.20889954,72.66516434,18.07311939],}} {{"COL)7": [68.00032082,39.91265109,83.47701751,42.71072597,33.54784094,94.63751895,3.364241739,0.792257736,78.63395232,70.8626348],}} {{"COL\n8": [77.80604836,61.08923308,70.70871195,99.33277829,79.77837072,56.28812485,34.03977847,13.40720489,87.71281052,64.80060217],}} {{"COL=9": [60.00505851,46.51367893,87.1346726,7.202332939,49.50378799,56.70949031,99.39792697,52.08074715,18.25891755,67.88110289],}} {{"COL\t10": [78.60259718,96.87558507,20.04134901,80.46408956,69.97610739,42.96954652,22.45733464,32.00411095,52.83023296,87.48870904],}} {{}, columns = ["COL 1","COL,2","COL;3","COL\{4","COL}5","COL(6","COL)7","COL\n8","COL=9","COL\t10"])}} {{df.to_parquet('invalid_columns_double.parquet')}} Here is a link to my databricks notebook [https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/5940072345564347/3863439224328194/623184285031795/latest.html] was (Author: toocoolblue2000): the schema is extracted from the same file, before materializing the data {{{{df = glue_context.read.schema(df.schema).parquet(}}{{'invalid_columns_double.parquet')}}}} {{{{ ^^^^^^^^^^^^^^^^^}}}} by schema I meant this. The file was written out from pandas dataframe. Here is a link to my databricks notebook https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/5940072345564347/3863439224328194/623184285031795/latest.html > Parquet with invalid chars on column name reads double as null when a clean > schema is applied > --------------------------------------------------------------------------------------------- > > Key: SPARK-34751 > URL: https://issues.apache.org/jira/browse/SPARK-34751 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 2.4.3, 3.1.1 > Environment: Pyspark 2.4.3 > AWS Glue Dev Endpoint EMR > Reporter: Nivas Umapathy > Priority: Major > Attachments: invalid_columns_double.parquet > > > I have a parquet file that has data with invalid column names on it. > [#Reference](https://issues.apache.org/jira/browse/SPARK-27442) Here is the > file attached with this ticket. > I tried to load this file with > {{df = glue_context.read.parquet('invalid_columns_double.parquet')}} > {{df = df.withColumnRenamed('COL 1', 'COL_1')}} > {{df = df.withColumnRenamed('COL,2', 'COL_2')}} > {{df = df.withColumnRenamed('COL;3', 'COL_3') }} > and so on. > Now if i call > {{df.show()}} > it throws this exception that is still pointing to the old column name. > {{pyspark.sql.utils.AnalysisException: 'Attribute name "COL 1" contains > invalid character(s) among " ,;{}()}} > {{n}} > {{t=". Please use alias to rename it.;'}} > > When i read about it in some blogs, there was suggestion to re-read the same > parquet with new schema applied. So i did > {{df = > glue_context.read.schema(df.schema).parquet(}}{{'invalid_columns_double.parquet')}} > > and it works, but all the data in the dataframe are null. The same works for > String datatypes > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org