[jira] [Commented] (SPARK-29047) use spark-submit Not a file: hdfs://

2019-10-11 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949182#comment-16949182
 ] 

Hyukjin Kwon commented on SPARK-29047:
--

Thanks for your response, [~我心凉凉的].

> use  spark-submit  Not a file: hdfs://
> --
>
> Key: SPARK-29047
> URL: https://issues.apache.org/jira/browse/SPARK-29047
> Project: Spark
>  Issue Type: Question
>  Components: Deploy, Spark Submit
>Affects Versions: 2.1.3
> Environment: spark1.3 ；hadoop2.6
>Reporter: bailin
>Priority: Critical
> Attachments: hadfs file is already existed.png
>
>
> when i submit a spark application : 
> {code:java}
> /spark-submit --class com.yto.log.SparkStatCleanJobYARN --name 
> TopNStatJobYARN --master yarn --executor-memory 1G  --num-executors 1 
> /home/hadoop/lib/sql-1.0-jar-with-dependencies.jar 
> hdfs://hadoop001:8020/clean 20161110{code}
> the hadfs file is already existed，but not recognition 
>  
> {code:java}
> 19/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 
> 219/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process 
> : 219/09/11 14:18:50 ERROR datasources.FileFormatWriter: Aborting job 
> null.java.io.IOException: Not a file: hdfs://hadoop001:8020/clean/day= at 
> org.apache.hadoop.mapred.FileInputFormat.getSplits
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29047) use spark-submit Not a file: hdfs://

2019-09-16 Thread bailin (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-29047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931013#comment-16931013
 ] 

bailin commented on SPARK-29047:


i solved, it's code problem.   Generate and read parquet,use 
"config("spark.sql.sources.partitionColumnTypeInference.enabled","false"  both

 

> use  spark-submit  Not a file: hdfs://
> --
>
> Key: SPARK-29047
> URL: https://issues.apache.org/jira/browse/SPARK-29047
> Project: Spark
>  Issue Type: Question
>  Components: Deploy, Spark Submit
>Affects Versions: 2.1.3
> Environment: spark1.3 ；hadoop2.6
>Reporter: bailin
>Priority: Critical
> Attachments: hadfs file is already existed.png
>
>
> when i submit a spark application : 
> {code:java}
> /spark-submit --class com.yto.log.SparkStatCleanJobYARN --name 
> TopNStatJobYARN --master yarn --executor-memory 1G  --num-executors 1 
> /home/hadoop/lib/sql-1.0-jar-with-dependencies.jar 
> hdfs://hadoop001:8020/clean 20161110{code}
> the hadfs file is already existed，but not recognition 
>  
> {code:java}
> 19/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 
> 219/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process 
> : 219/09/11 14:18:50 ERROR datasources.FileFormatWriter: Aborting job 
> null.java.io.IOException: Not a file: hdfs://hadoop001:8020/clean/day= at 
> org.apache.hadoop.mapred.FileInputFormat.getSplits
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org