Github user sadhen commented on the issue:
https://github.com/apache/spark/pull/14638
yes
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
@sadhen Your solution works well without any code change, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user sadhen commented on the issue:
https://github.com/apache/spark/pull/14638
Since this pr won't be accepted. My solution is:
``` sql
# create table using spark-csv
CREATE TABLE tmp.cars
USING com.databricks.spark.csv
OPTIONS (path "/tmp/cars.csv", header
Github user sadhen commented on the issue:
https://github.com/apache/spark/pull/14638
Also a useful feature for me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user Derek-Lin commented on the issue:
https://github.com/apache/spark/pull/14638
Sadly, it's a useful feature which i want.
i want to retreive HDFS tsv files via spark sql jdbc interface which
defined as hive table
---
If your project is set up for it, you can reply to
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you all so much for reviewing and discussion on this issue since Aug
14th.
Since we focused this fully both here and in Spark mailing list and it has
been 4 months already, I'll
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Do you mean to replace the current whole `TableReader.scala` which was
introduced in SPARK-1251 ? I guessed Spark chose this direct access approach
for the performance issue at that time.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14638
Is there any high-level API that we can use to read hive tables? The
current hive table reader is so low-level that we have to support features like
`skip.header.line.count`, and I think it
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you so much, @jamartinh , @srowen , @HyukjinKwon , and @gatorsmile .
We can distinguish the two existing problems separately here.
First, **a)** Spark returns incorrect
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
@jamartinh I got your point. Hive tables can be read/wrote by both Hive and
Spark. Ideally, we should respect all the features supported by Hive. That
means, if the metadata of an existing Hive
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14638
Up to my knowledge, currently, CSV datasource does supports to skip a
(only) single line with `header` option and also `comment` option to skip the
lines started with the character given to
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14638
I am really on the fence about it. I can see the argument that it's useful
to support, and not hard to support. But also that this is just one of many
Hive features that one could support, and we
Github user jamartinh commented on the issue:
https://github.com/apache/spark/pull/14638
My humble comment is simply that an external Hive table still is a Hive
Table, and as such it should be transparent for whichever user how to work
with it, even for spark users like me. The
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Now, I think we have been considered almost pros and cons here. Thank you
all for reviewing and giving opinions for this issue.
I value the committers' decision and feel the worry
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you for opinion, @HyukjinKwon and @cloud-fan.
BTW, our CSV data sources does not support multiple line headers so far.
---
If your project is set up for it, you can reply to
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14638
@HyukjinKwon yea for Hive this feature is useful, as the only workaround is
running a ETL program to remove the headers manually. However, in Spark SQL,
our CSV data source already support
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14638
I know votes and stars might not be an appropriate standard for the
popularity of a feature but I found another stackoverflow question -
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14638
I think this is really an uncommon hive feature, and users can work around
it by creating an external data source table on the CSV files.
And another concern is, we can't simply call some
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Yes. Right, that is possible, but it's a different use case.
This PR supports the followings:
- The **existing Hive table** with
`TBLPROPERTIES(skip.header.line.count=1)` will
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
That list is not exhaustive, but should we follow what we did for CSV,
instead of using this hidden Hive feature? Should we use `OPTIONS` instead of
`TBLPROPERTIES`?
---
If your project is set
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you for the investigation. Is the Hive table property list exhaustive?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
Also cc @cloud-fan @hvanhovell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
In DDL for creating data source tables, `OPTIONS` is being used for
table-level parameters. In the near future, we need to combine DDL for data
source tables and Hive serde tables. I am not sure
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
In the Hive document, this is not included.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable
The list does not include it:
```
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Actually, it's a Spark issue SPARK-11374, isn't it?
So, the remaining question should be `the Spark user do want this`.
For this, I'll send an email to user and dev mailing list.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Ah, if that is the question, I knew the customers using this and requesting
this. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
That is my question. I do not know the answer. Based on the comment in the
Hive JIRA, I am doubting how many Hive users are using it.
---
If your project is set up for it, you can reply to
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Why do you think `Hive user are not using this feature`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
If Hive users are not using this feature, should we still use the table
property as the interface to do it?
Asking users to change the table property looks hacky to me.
---
If your
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Hi, @gatorsmile .
Do you think it's valid comment for this implementation? Why **'this forces
entire input into a single split'**? We are not doing such a thing.
For me, it looks like
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14638
In the original Hive JIRA of this feature:
https://issues.apache.org/jira/browse/HIVE-5795
See the latest reply from `Sergey Shelukhin`.
> This forces the entire input into a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69830/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69830 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69830/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69830 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69830/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69816/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69816 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69816/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69816 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69816/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you for review, @mridulm .
I updated the PR by using `isAssignableFrom`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69796/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69796 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69796/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69796 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69796/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Hi, @rxin , @srowen , @yhuai .
If there is something for me to do more for this issue, please let me know.
---
If your project is set up for it, you can reply to this email and have your
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Could you review this again please when you have sometime, @rxin ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Hi, @rxin .
Do you want me to prepare something based on `InputFileBlockHolder` of
#16133 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
@rxin
Could you give me any advice for this issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Hi, @rxin .
Here is the corresponding HIve implementation.
**getNextRow**
*
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
I didn't read Hive code part for this. I referenced the Spark csv part code.
But, I'll search the corresponding one in Hive master branch tomorrow since
you requested.
---
If your
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14638
@dongjoon-hyun can you show a link to how Hive implements this? I'm just
surprised that this is a general feature rather than something that is format
specific.
---
If your project is set up for
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69539/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69539 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69539/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69539 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69539/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
@rxin .
I updated the title and description of this PR.
For the test case, I think the current test case covers enough the
functionality.
---
If your project is set up for it, you
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Ah, I see.
Then, I'll try to update PR descriptions or testcases more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14638
The way the description is written implies that there is something special
about external tables or tables with location specified. I'm just confused here.
---
If your project is set up for it, you
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
@rxin .
I didn't say this is only a problem for external table.
I simply focused on SPARK-11374 use case here.
What use case are you considering when you mentioned **managed
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14638
@dongjoon-hyun
can you explain why this is only a problem for external table but not
managed tables?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you for approving, @srowen !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69362/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69362 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69362/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69362 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69362/consoleFull)**
for PR 14638 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69334/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69334 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69334/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #69334 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69334/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68633/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #68633 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68633/consoleFull)**
for PR 14638 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68628/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #68628 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68628/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #68633 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68633/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you for review, @srowen !
Yes. The issue was `the question here is whether the change belongs here or
elsewhere`, and I couldn't find better place.
---
If your project is set up
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #68628 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68628/consoleFull)**
for PR 14638 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14638
Thank you, @jamartinh . I hope so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user jamartinh commented on the issue:
https://github.com/apache/spark/pull/14638
This is necessary, please add this to a release !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68460/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #68460 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68460/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #68460 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68460/consoleFull)**
for PR 14638 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67426/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #67426 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67426/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #67426 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67426/consoleFull)**
for PR 14638 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66845/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #66845 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66845/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #66845 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66845/consoleFull)**
for PR 14638 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66493/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #66493 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66493/consoleFull)**
for PR 14638 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14638
**[Test build #66493 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66493/consoleFull)**
for PR 14638 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66228/
Test PASSed.
---
1 - 100 of 156 matches
Mail list logo
