Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@stczwd, I agree with @mccheah. Tables are basically named data sets.
Whether they support batch, micro-batch streaming, or continuous streaming is
determined by checking whether they implement
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21306
@stczwd that's my understanding yeah. Others can feel free to correct me
otherwise.
---
-
To unsubscribe, e-mail:
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/21306
@mccheah you mean the tables user created do not distinguish between stream
and batch, but only when they are actually read from it?
---
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21306
@stczwd my understanding here is that a table isn't a streaming table or a
batch table, but rather that a table points to data that can either be scanned
in stream or in batch, and that the table
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/21306
> @stczwd, thanks for taking a look at this. What are the differences
between batch and stream DDL that you think will come up?
1. Source needs to be defined for stream table
2. Stream
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21306
A general question. How to use this catalog API to implement the Hive
metastore? Is it doable?
---
-
To unsubscribe, e-mail:
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@stczwd, thanks for taking a look at this. What are the differences between
batch and stream DDL that you think will come up?
---
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/21306
@rdblue Have you considered about stream table API? It may have some
differences between batch table ddl and stream table ddl.
---
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@felixcheung, we're waiting on more reviews and a community decision about
how to pass partition transforms.
For passing transforms, I think the most reasonable compromise is to go
with a
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21306
where are we on this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@tigerquoll, the proposal isn't to make partitions part of table
configuration. It is to make the partitioning scheme part of the table
configuration. How sources choose to handle individual
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21306
So Kudu range partitions support arbitrary sized partition intervals, like
the example below, where the first and last range partition are six months in
size, but the middle partition is one
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21306
Sure,
I am looking at the point of view of supporting Kudu. Check out
https://kudu.apache.org/docs/schema_design.html#partitioning for some of the
details. In particular
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
> Can we support column range partition predicates please?
This has an "apply" transform for passing other functions directly through,
so that may help if you have additional transforms that
Github user tigerquoll commented on the issue:
https://github.com/apache/spark/pull/21306
Can we support column range partition predicates please?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94982/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94982 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94982/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94982 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94982/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94918 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94918/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94918/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94918 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94918/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94917 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94917/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94917/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94917 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94917/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94906/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94906 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94906/testReport)**
for PR 21306 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94906 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94906/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94825/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94825 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94825/testReport)**
for PR 21306 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@rxin, I've updated this to use a new interface, `PartitionTransform`,
instead of `Expression`. This is used to pass well-known transformations when
creating tables, like `Filter` is used to pass a
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94825 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94825/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94773/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94773 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94773/testReport)**
for PR 21306 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #94773 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94773/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@cloud-fan, on the dev thread about 2.4 you talked about getting this PR
in. What do we need to do next?
I can call a vote on the SPIP if you think that's ready. I just bumped the
thread.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
#21888 demonstrates how to add a `TableIdentifier` with a catalog element,
`CatalogTableIdentifier` and how to safely migrate from the old identifier to
the new one with catalog.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93605/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93605 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93605/testReport)**
for PR 21306 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93605 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93605/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93581 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93581/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93581/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93581 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93581/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93568/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93568 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93568/testReport)**
for PR 21306 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93568 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93568/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93567 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93567/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93567/
Test FAILed.
---
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21306
@marmbrus, @cloud-fan, @gatorsmile, I've updated this PR to use reflection
to instantiate catalogs. This allows implementations to provide named catalogs
(and reuse implementations) and configure
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93567 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93567/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93556/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93556 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93556/testReport)**
for PR 21306 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21306
**[Test build #93556 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93556/testReport)**
for PR 21306 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
86 matches
Mail list logo
