Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
See https://issues.apache.org/jira/browse/SPARK-6351 ~ Jonathan From: Shuai Zheng szheng.c...@gmail.commailto:szheng.c...@gmail.com Date: Monday, March 16, 2015 at 11:46 AM To: user@spark.apache.orgmailto:user@spark.apache.org user@spark.apache.orgmailto:user@spark.apache.org Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn’t work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile(s3n:///test/2.parq ) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:///file:///\\ And same test work with spark-shell under 1.2.1 Regards, Shuai
sqlContext.parquetFile doesn't work with s3n in version 1.3.0
Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn't work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile(s3n:///test/2.parq ) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:/// file:///\\ And same test work with spark-shell under 1.2.1 Regards, Shuai
RE: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
I see, but this is really a. big issue. anyway for me to work around? I try to set the fs.default.name = s3n, but looks like it doesn't work. I must upgrade to 1.3.0 because I face the package incompatible issue in 1.2.1, and if I must patch something, I rather go with latest version. Regards, Shuai From: Kelly, Jonathan [mailto:jonat...@amazon.com] Sent: Monday, March 16, 2015 2:54 PM To: Shuai Zheng; user@spark.apache.org Subject: Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 See https://issues.apache.org/jira/browse/SPARK-6351 ~ Jonathan From: Shuai Zheng szheng.c...@gmail.com Date: Monday, March 16, 2015 at 11:46 AM To: user@spark.apache.org user@spark.apache.org Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn't work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile(s3n:///test/2.parq ) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:/// file:///\\ And same test work with spark-shell under 1.2.1 Regards, Shuai
Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
We will be including this fix in Spark 1.3.1 which we hope to make in the next week or so. On Mon, Mar 16, 2015 at 12:01 PM, Shuai Zheng szheng.c...@gmail.com wrote: I see, but this is really a… big issue. anyway for me to work around? I try to set the fs.default.name = s3n, but looks like it doesn’t work. I must upgrade to 1.3.0 because I face the package incompatible issue in 1.2.1, and if I must patch something, I rather go with latest version. Regards, Shuai *From:* Kelly, Jonathan [mailto:jonat...@amazon.com] *Sent:* Monday, March 16, 2015 2:54 PM *To:* Shuai Zheng; user@spark.apache.org *Subject:* Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 See https://issues.apache.org/jira/browse/SPARK-6351 ~ Jonathan *From: *Shuai Zheng szheng.c...@gmail.com *Date: *Monday, March 16, 2015 at 11:46 AM *To: *user@spark.apache.org user@spark.apache.org *Subject: *sqlContext.parquetFile doesn't work with s3n in version 1.3.0 Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn’t work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile(s3n:///test/2.parq ) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:/// And same test work with spark-shell under 1.2.1 Regards, Shuai
