[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24290001
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -315,9 +335,17 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
}
}
+/**
+ * when the soi and deserializer.getObjectInspector is equal,
+ * we will get `IdentityConverter`,which mean it won't convert the
+ * value when schema match
+ */
+val partTblObjectInspectorConverter =
ObjectInspectorConverters.getConverter(
+ deserializer.getObjectInspector, soi)
+
// Map each tuple to a row object
iterator.map { value =
- val raw = deserializer.deserialize(value)
+ val raw =
partTblObjectInspectorConverter.convert(deserializer.deserialize(value))
--- End diff --
@jeanlyn Thank you for the explanation offline, after reading the source
code of Hive `ObjectInspector Converter`, I realize the `Converter` only
support the `SettableObjectInspector`, that's why we need the `ConvertedOI`
also.
I've also tested your code by adding some of my thought at
https://github.com/chenghao-intel/spark/commit/955cd38cab53c9db8bbba3b9ea84a326cc86ac43
Feel free to take it's helpful. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73368110 Thank you @jeanlyn for the explanation offline, I've added some comments also. And I also have some idea to simplify the code, just take it if you feel reasonable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73377819 [Test build #27008 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27008/consoleFull) for PR 4289 at commit [`c879aa1`](https://github.com/apache/spark/commit/c879aa1d68ef41ca95ce2084326a6f14a00deb4f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73377821 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27008/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24290012
--- Diff:
sql/hive/v0.12.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala ---
@@ -242,6 +242,14 @@ private[hive] object HiveShim {
}
}
+ // make getConvertedOI compatible between 0.12.0 and 0.13.1
--- End diff --
The comment seems not necessary, as all of the methods in this file are for
the shim purpose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24290013
--- Diff:
sql/hive/v0.13.1/src/main/scala/org/apache/spark/sql/hive/Shim13.scala ---
@@ -397,7 +398,12 @@ private[hive] object HiveShim {
Decimal(hdoi.getPrimitiveJavaObject(data).bigDecimalValue(),
hdoi.precision(), hdoi.scale())
}
}
-
+
+ // make getConvertedOI compatible between 0.12.0 and 0.13.1
--- End diff --
Nit: Same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73380175 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27010/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73380174 [Test build #27010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27010/consoleFull) for PR 4289 at commit [`1e8b30c`](https://github.com/apache/spark/commit/1e8b30cea76d1f0594beb37a47252118123b649d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24290009
--- Diff:
sql/hive/v0.12.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala ---
@@ -242,6 +242,14 @@ private[hive] object HiveShim {
}
}
+ // make getConvertedOI compatible between 0.12.0 and 0.13.1
+ def getConvertedOI(inputOI: ObjectInspector,
+ outputOI: ObjectInspector,
+ equalsCheck: java.lang.Boolean =
--- End diff --
let's remove the parameter, to keep it as the same method signature within
the `HiveShim` 0.13.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73377784 [Test build #27008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27008/consoleFull) for PR 4289 at commit [`c879aa1`](https://github.com/apache/spark/commit/c879aa1d68ef41ca95ce2084326a6f14a00deb4f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24290017
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -264,15 +268,31 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
* @param nonPartitionKeyAttrs Attributes that should be filled together
with their corresponding
* positions in the output schema
* @param mutableRow A reusable `MutableRow` that should be filled
+ * @param convertdeserializer The `Deserializer` covert the
`deserializer`
* @return An `Iterator[Row]` transformed from `iterator`
*/
def fillObject(
iterator: Iterator[Writable],
deserializer: Deserializer,
nonPartitionKeyAttrs: Seq[(Attribute, Int)],
- mutableRow: MutableRow): Iterator[Row] = {
+ mutableRow: MutableRow,
+ convertdeserializer: Option[Deserializer] = None): Iterator[Row] = {
--- End diff --
variable name should be in camel style.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73378452 [Test build #27010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27010/consoleFull) for PR 4289 at commit [`1e8b30c`](https://github.com/apache/spark/commit/1e8b30cea76d1f0594beb37a47252118123b649d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user jeanlyn commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73459718 Thanks @chenghao-intel for review and suggestions!I take some of your advises to simplify the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73453424 [Test build #27066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27066/consoleFull) for PR 4289 at commit [`d6c93c5`](https://github.com/apache/spark/commit/d6c93c54eadfc216e29377920f2c4937ae73cd4e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73453428 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27066/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73449265 [Test build #27066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27066/consoleFull) for PR 4289 at commit [`d6c93c5`](https://github.com/apache/spark/commit/d6c93c54eadfc216e29377920f2c4937ae73cd4e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24222130
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -315,9 +335,17 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
}
}
+/**
+ * when the soi and deserializer.getObjectInspector is equal,
+ * we will get `IdentityConverter`,which mean it won't convert the
+ * value when schema match
+ */
+val partTblObjectInspectorConverter =
ObjectInspectorConverters.getConverter(
+ deserializer.getObjectInspector, soi)
+
// Map each tuple to a row object
iterator.map { value =
- val raw = deserializer.deserialize(value)
+ val raw =
partTblObjectInspectorConverter.convert(deserializer.deserialize(value))
--- End diff --
There logic here is a little confusing for me.
As we already have the `converter` here, probably we don't need to call the
`getConvertedOI`, and the `soi` should be the expected output
`ObjectInspector`, which supposed to be the output object inspector of from the
table deserializer. right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24222165
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -315,9 +335,17 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
}
}
+/**
+ * when the soi and deserializer.getObjectInspector is equal,
+ * we will get `IdentityConverter`,which mean it won't convert the
+ * value when schema match
+ */
+val partTblObjectInspectorConverter =
ObjectInspectorConverters.getConverter(
+ deserializer.getObjectInspector, soi)
+
// Map each tuple to a row object
iterator.map { value =
- val raw = deserializer.deserialize(value)
+ val raw =
partTblObjectInspectorConverter.convert(deserializer.deserialize(value))
--- End diff --
We can discuss that offline if you feel confusing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24221959
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -264,15 +268,31 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
* @param nonPartitionKeyAttrs Attributes that should be filled together
with their corresponding
* positions in the output schema
* @param mutableRow A reusable `MutableRow` that should be filled
+ * @param convertdeserializer The `Deserializer` covert the
`deserializer`
* @return An `Iterator[Row]` transformed from `iterator`
*/
def fillObject(
iterator: Iterator[Writable],
deserializer: Deserializer,
nonPartitionKeyAttrs: Seq[(Attribute, Int)],
- mutableRow: MutableRow): Iterator[Row] = {
+ mutableRow: MutableRow,
+ convertdeserializer: Option[Deserializer] = None): Iterator[Row] = {
--- End diff --
Change the `convertdeserializer` to `outputStructObjectInspector`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user jeanlyn commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-73176093 hi,@chenghao-intel @marmbrus any suggestions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72988937 [Test build #26820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26820/consoleFull) for PR 4289 at commit [`2a91a87`](https://github.com/apache/spark/commit/2a91a87d27e31497ebefd2ad80531c0017fafbc5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72993874 [Test build #26820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26820/consoleFull) for PR 4289 at commit [`2a91a87`](https://github.com/apache/spark/commit/2a91a87d27e31497ebefd2ad80531c0017fafbc5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72993877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26820/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24138028
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -315,9 +335,23 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
}
}
+val partTblObjectInspectorConverter =
ObjectInspectorConverters.getConverter(
+ deserializer.getObjectInspector, soi)
+
// Map each tuple to a row object
iterator.map { value =
- val raw = deserializer.deserialize(value)
+ val raw = convertdeserializer match {
--- End diff --
Thanks for remind
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24139046
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -264,15 +268,31 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
* @param nonPartitionKeyAttrs Attributes that should be filled together
with their corresponding
* positions in the output schema
* @param mutableRow A reusable `MutableRow` that should be filled
+ * @param convertdeserializer The `Deserializer` covert the
`deserializer`
* @return An `Iterator[Row]` transformed from `iterator`
*/
def fillObject(
iterator: Iterator[Writable],
deserializer: Deserializer,
nonPartitionKeyAttrs: Seq[(Attribute, Int)],
- mutableRow: MutableRow): Iterator[Row] = {
+ mutableRow: MutableRow,
+ convertdeserializer: Option[Deserializer] = None): Iterator[Row] = {
--- End diff --
But the `val soi` also need a convert deserializer when the schema doesn't
match
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24139524
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -264,15 +268,31 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
* @param nonPartitionKeyAttrs Attributes that should be filled together
with their corresponding
* positions in the output schema
* @param mutableRow A reusable `MutableRow` that should be filled
+ * @param convertdeserializer The `Deserializer` covert the
`deserializer`
* @return An `Iterator[Row]` transformed from `iterator`
*/
def fillObject(
iterator: Iterator[Writable],
deserializer: Deserializer,
nonPartitionKeyAttrs: Seq[(Attribute, Int)],
- mutableRow: MutableRow): Iterator[Row] = {
+ mutableRow: MutableRow,
+ convertdeserializer: Option[Deserializer] = None): Iterator[Row] = {
--- End diff --
OK, you're right, forget about my comment above. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-7280 In general I think the change looks reasonable to me, and we'd better use the Hive `ObjectConverter` directly, and some of the code can be cleaner. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24057600
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -315,9 +335,23 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
}
}
+val partTblObjectInspectorConverter =
ObjectInspectorConverters.getConverter(
+ deserializer.getObjectInspector, soi)
+
// Map each tuple to a row object
iterator.map { value =
- val raw = deserializer.deserialize(value)
+ val raw = convertdeserializer match {
--- End diff --
In general, we'd better not to do the pattern matching within the iterator,
and we can do that like:
`
xx match {
case xxx = iterator.map { ... }
case yyy = iterator.map { ... }
}
```
For this case, as I shown above, if we passed the converter directly into
`fillObject`, I don't think we need the pattern match here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24057968
--- Diff:
sql/hive/v0.12.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala ---
@@ -242,6 +242,11 @@ private[hive] object HiveShim {
}
}
+ // make getConvertedOI compatible between 0.12.0 and 0.13.1
+ def getConvertedOI(inputOI: ObjectInspector, outputOI: ObjectInspector):
ObjectInspector = {
+ObjectInspectorConverters.getConvertedOI(inputOI, outputOI, new
java.lang.Boolean(true))
--- End diff --
Just `true` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24058262
--- Diff:
sql/hive/v0.12.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala ---
@@ -242,6 +242,11 @@ private[hive] object HiveShim {
}
}
+ // make getConvertedOI compatible between 0.12.0 and 0.13.1
+ def getConvertedOI(inputOI: ObjectInspector, outputOI: ObjectInspector):
ObjectInspector = {
--- End diff --
And use the `ObjectInspectorConverters.getConverter` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72671869 Sorry for the late reply @jeanlyn ! I think it's a bug of Hive DDL, which probably was resolved in Hive 0.14 / 0.15, and I am not sure if we really want to fix that in Spark SQL. @yhuai , do you have any comment on this? However, in this particular case, another work around in your product: 1) Rename the existed table; 2) Create a new table with schema you altered, and also the partitions. 3) Manually move the previous data into the new table folder from the HDFS. 4) Drop the old table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24010127
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala
---
@@ -172,4 +172,19 @@ class InsertIntoHiveTableSuite extends QueryTest {
sql(DROP TABLE hiveTableWithStructValue)
}
+
+ test(SPARK-5498:partition schema does not match table schema){
+val testData = TestHive.sparkContext.parallelize(
+ (1 to 10).map(i = TestData(i, i.toString)))
+testData.registerTempTable(testData)
+val tmpDir = Files.createTempDir()
+sql(sCREATE TABLE table_with_partition(key int,value string)
PARTITIONED by (ds string) location '${tmpDir.toURI.toString}' )
+sql(INSERT OVERWRITE TABLE table_with_partition partition (ds='1')
SELECT key,value FROM testData)
+sql(ALTER TABLE table_with_partition CHANGE COLUMN key key BIGINT)
--- End diff --
I just checked the [Hive
Document](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable)
It says:
`The CASCADE|RESTRICT clause is available in Hive 0.15.0. ALTER TABLE
CHANGE COLUMN with CASCADE command changes the columns of a table's metadata,
and cascades the same change to all the partition metadata. RESTRICT is the
default, limiting column change only to table metadata.`
I guess in Hive 0.13.1, when table schema changed via `alter table`, only
the table meta data will be updated, can you double check if above query works
for Hive 0.13.1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24013566
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertIntoHiveTableSuite.scala
---
@@ -172,4 +172,19 @@ class InsertIntoHiveTableSuite extends QueryTest {
sql(DROP TABLE hiveTableWithStructValue)
}
+
+ test(SPARK-5498:partition schema does not match table schema){
+val testData = TestHive.sparkContext.parallelize(
+ (1 to 10).map(i = TestData(i, i.toString)))
+testData.registerTempTable(testData)
+val tmpDir = Files.createTempDir()
+sql(sCREATE TABLE table_with_partition(key int,value string)
PARTITIONED by (ds string) location '${tmpDir.toURI.toString}' )
+sql(INSERT OVERWRITE TABLE table_with_partition partition (ds='1')
SELECT key,value FROM testData)
+sql(ALTER TABLE table_with_partition CHANGE COLUMN key key BIGINT)
--- End diff --
I check this query in Hive 0.11 is OK,I will check this query in Hive
0.13.1 later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user jeanlyn commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72680801 Thanks @chenghao-intel for review and suggestions! We want to replace some hive sql to spark-sql in our production environment,so I use some sql in our production environmeng which running in hive-0.12 to test spark-sql and i found this issue,so i think make spark-sql to more compatible is well for popularized,and i will test the points @chenghao-intel listed both in hive and spark-sql. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72773457 Oh, @jeanlyn, I've also tested that in Hive 0.13, seems it works. Hive will do the data type converting if it realizes the partition schema is not the same as the table schema. Your change seems reasonable. I will review the rest of the code, hope fully we can catch up the 0.13 release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4289#discussion_r24057467
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -264,15 +268,31 @@ private[hive] object HadoopTableReader extends
HiveInspectors {
* @param nonPartitionKeyAttrs Attributes that should be filled together
with their corresponding
* positions in the output schema
* @param mutableRow A reusable `MutableRow` that should be filled
+ * @param convertdeserializer The `Deserializer` covert the
`deserializer`
* @return An `Iterator[Row]` transformed from `iterator`
*/
def fillObject(
iterator: Iterator[Writable],
deserializer: Deserializer,
nonPartitionKeyAttrs: Seq[(Attribute, Int)],
- mutableRow: MutableRow): Iterator[Row] = {
+ mutableRow: MutableRow,
+ convertdeserializer: Option[Deserializer] = None): Iterator[Row] = {
--- End diff --
Instead of passing the `deserializer`, how about take the `converter` as
the argument? By the way, I think Hive provides the `IdentityConverter`, which
mean we can make the parameter as ObjectInspectorConverters.Converter, not
necessary wrapped by `Option`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72587496 [Test build #26592 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26592/consoleFull) for PR 4289 at commit [`7470901`](https://github.com/apache/spark/commit/74709017763d18f6ecdefbdf512500c2538b2154). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72587499 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26592/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72596283 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26602/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72596740 [Test build #26611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26611/consoleFull) for PR 4289 at commit [`12d800d`](https://github.com/apache/spark/commit/12d800d0904fba3564caba7613a35caa6fc812ce). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72587243 [Test build #26592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26592/consoleFull) for PR 4289 at commit [`7470901`](https://github.com/apache/spark/commit/74709017763d18f6ecdefbdf512500c2538b2154). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72591570 [Test build #26602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26602/consoleFull) for PR 4289 at commit [`63d170a`](https://github.com/apache/spark/commit/63d170a633304c63ff55d2be399442d9784cfe8d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72604052 [Test build #26611 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26611/consoleFull) for PR 4289 at commit [`12d800d`](https://github.com/apache/spark/commit/12d800d0904fba3564caba7613a35caa6fc812ce). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72604056 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26611/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72596278 [Test build #26602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26602/consoleFull) for PR 4289 at commit [`63d170a`](https://github.com/apache/spark/commit/63d170a633304c63ff55d2be399442d9784cfe8d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Column extends DataFrame with ExpressionApi ` * `class ColumnName(name: String) extends IncomputableColumn(name) ` * `trait DataFrame extends DataFrameSpecificApi with RDDApi[Row] ` * `class GroupedDataFrame protected[sql](df: DataFrameImpl, groupingExprs: Seq[Expression])` * ` protected[sql] class QueryExecution(val logical: LogicalPlan) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72400702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26484/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72400699 [Test build #26484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26484/consoleFull) for PR 4289 at commit [`b1527d5`](https://github.com/apache/spark/commit/b1527d58349ccdc0b986705b93d7658822211994). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72397218 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72397279 /cc @chenghao-intel --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72397455 [Test build #26479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26479/consoleFull) for PR 4289 at commit [`adfc7de`](https://github.com/apache/spark/commit/adfc7defb278667d0c27c6128b00339bb8d52bb1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72397543 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26479/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72397542 [Test build #26479 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26479/consoleFull) for PR 4289 at commit [`adfc7de`](https://github.com/apache/spark/commit/adfc7defb278667d0c27c6128b00339bb8d52bb1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72400374 [Test build #26484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26484/consoleFull) for PR 4289 at commit [`b1527d5`](https://github.com/apache/spark/commit/b1527d58349ccdc0b986705b93d7658822211994). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72398464 [Test build #26481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26481/consoleFull) for PR 4289 at commit [`10744ca`](https://github.com/apache/spark/commit/10744cac4cc014216241fc97c6495924cc51fdbc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72398796 [Test build #26481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26481/consoleFull) for PR 4289 at commit [`10744ca`](https://github.com/apache/spark/commit/10744cac4cc014216241fc97c6495924cc51fdbc). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72398798 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26481/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72406797 [Test build #26489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26489/consoleFull) for PR 4289 at commit [`afc7da5`](https://github.com/apache/spark/commit/afc7da53be4b7bcb9cd5ce8d72b6855544b96596). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72410648 [Test build #26489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26489/consoleFull) for PR 4289 at commit [`afc7da5`](https://github.com/apache/spark/commit/afc7da53be4b7bcb9cd5ce8d72b6855544b96596). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class Rating[@specialized(Int, Long) ID](user: ID, item: ID, rating: Float)` * `class StandardScalerModel (` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5498][SQL]fix bug when query the data w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4289#issuecomment-72410655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26489/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
