Hello, Stuart.
Do you need any assistance with this task from me or other community member?
В Вт, 04/09/2018 в 19:03 +0300, Nikolay Izhikov пишет:
> Hello, Stuart.
>
> Sorry for the silence.
>
> I was swamped the last couple of days.
>
> I think you can go forward and implement suggested
Hello, Stuart.
Sorry for the silence.
I was swamped the last couple of days.
I think you can go forward and implement suggested solution.
I'm -0 with it.
So no block from my side, but I'm still no happy with abstractions :).
В Пн, 03/09/2018 в 09:35 +0100, Stuart Macdonald пишет:
> Nikolay,
Nikolay, Val, it would be good if we could reach agreement here so that I
can make the necessary modifications before the 2.7 cutoff.
Nikolay - would you be comfortable if I went ahead and made database=schema?
Stuart.
On Mon, Aug 27, 2018 at 10:22 PM Valentin Kulichenko <
Hi Nikolay,
I think it's actually pretty unfortunate that Spark uses term "database"
here, as it essentially refers to a schema in my view. Usually, database is
something you create a physical connection to, and connection is bind to
that database. To connect to another database you need to
Igniters,
Personally, I don't like the solution with database == schema name.
1. I think we should try to use the right abstractions.
schema == database doesn't sound right for me.
Do you want to answer to all of our users something like that:
- "How I can change Ignite SQL schema?"
- "This
I'll go ahead and make the changes to represent the schema name as the
database name for the purposes of the Spark catalog.
If anyone knows of an existing way to list all available schemata within an
Ignite instance please let me know, otherwise the first task will be
creating that mechanism.
Nikolay,
If there are multiple configuration in XML, IgniteContext will always use
only one of them. Looks like current approach simply doesn't work. I
propose to report schema name as 'database' in Spark. If there are multiple
clients, you would create multiple sessions and multiple catalogs.
Hello, Valentin.
> catalog exist in scope of a single IgniteSparkSession> (and therefore single
> IgniteContext and single Ignite instance)?
Yes.
Actually, I was thinking about use case when we have several Ignite
configuration in one XML file.
Now I see, may be this is too rare use-case to
Nikolay,
Whatever we decide on would be right :) Basically, we need to answer this
question: does the catalog exist in scope of a single IgniteSparkSession
(and therefore single IgniteContext and single Ignite instance)? In other
words, in case of a rare use case when a single Spark application
Hello, Valentin.
> I believe we should get rid of this logic and use Ignite schema name as
> database name in Spark's catalog.
When I develop Ignite integration with Spark Data Frame I use following
abstraction described by Vladimir Ozerov:
"1) Let's consider Ignite cluster as a single
Hi Val, yes that's correct. I'd be happy to make the change to have the
database reference the schema if Nikolay agrees. (I'll first need to do a
bit of research into how to obtain the list of all available schemata...)
Thanks,
Stuart.
On Tue, Aug 21, 2018 at 9:43 PM, Valentin Kulichenko <
Stuart,
Thanks for pointing this out, I was not aware that we use Spark database
concept this way. Actually, this confuses me a lot. As far as I understand,
catalog is created in the scope of a particular IgniteSparkSession, which
in turn is assigned to a particular IgniteContext and therefore
Nikolay, Val,
The JDBC Spark datasource[1] -- as far as I can tell -- has no
ExternalCatalog implementation, it just uses the database specified in the
JDBC URL. So I don't believe there is any way to call listTables() or
listDatabases() for JDBC provider.
The Hive ExternalCatalog[2] makes the
Hello, Stuart.
Can you do some research and find out how schema is handled in Data Frames for
a regular RDBMS such as Oracle, MySQL, etc?
В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> Stuart, Nikolay,
>
> I see that the 'Table' class (returned by listTables method) has a
Stuart, Nikolay,
I see that the 'Table' class (returned by listTables method) has a
'database' field. Can we use this one to report schema name?
In any case, I think we should look into how this is done in data source
implementations for other databases. Any relational database has a notion
of
Hello, Stuart.
Personally, I think we should change current tables naming and return table in
form of `schema.table`.
Valentin, could you share your opinion?
В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет:
> Igniters,
>
> While reviewing the changes for IGNITE-9228 [1,2], Nikolay and
16 matches
Mail list logo
