[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72614700 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26639/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72614694 [Test build #26639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26639/consoleFull) for PR 4325 at commit [`e46735c`](https://github.com/apache/spark/commit/e46735c612879bb46317efe73155b4611bb51afc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/4325#discussion_r23988701
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DateUtils.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.sql.Date
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * helper function to convert between Int value of days since 1970-01-01
and java.sql.Date
+ */
+object DateUtils {
--- End diff --
If it's not necessary, just do not make it public, or we can not change it
anymore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4325#discussion_r23988264
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DateUtils.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.sql.Date
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * helper function to convert between Int value of days since 1970-01-01
and java.sql.Date
+ */
+object DateUtils {
--- End diff --
I think this could be useful even outside of spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72613604 [Test build #26639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26639/consoleFull) for PR 4325 at commit [`e46735c`](https://github.com/apache/spark/commit/e46735c612879bb46317efe73155b4611bb51afc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/4325#discussion_r23990278
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DateUtils.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.sql.Date
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * helper function to convert between Int value of days since 1970-01-01
and java.sql.Date
+ */
+object DateUtils {
--- End diff --
Code gen need that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72615808 [Test build #26642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26642/consoleFull) for PR 4325 at commit [`096e20d`](https://github.com/apache/spark/commit/096e20d5de068157910372a03a6face9edc829e6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72615722 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4325 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72625660 [Test build #26642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26642/consoleFull) for PR 4325 at commit [`096e20d`](https://github.com/apache/spark/commit/096e20d5de068157910372a03a6face9edc829e6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72625668 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26642/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/4325#discussion_r23990833
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DateUtils.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.sql.Date
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * helper function to convert between Int value of days since 1970-01-01
and java.sql.Date
+ */
+object DateUtils {
--- End diff --
That make sense, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3732 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-72563968 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/4325 [SPARK-4508] [SQL] build native date type to conform behavior to Hive The previous #3732 is reverted due to some test failure. Have fixed that. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian-wang/spark datenative Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4325.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4325 commit 374abd52b422642e1ef73d617df9aa9d395803b7 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-15T04:49:36Z spark native date type support commit d6715fcac2a4c5a32a6b8c11f84567c675de1892 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-27T06:40:55Z refactoring Date as Primitive Int internally commit 024c9a6d37da9930b2c8d57cf4c01543ef0080e2 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-27T07:22:22Z clean some import order commit f0005b166a705f7b1c52960b72c4ff29d010e5ff Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-28T07:19:50Z add date in sql parser and java type conversion commit c37832bc3a48493639b7a74d3277c11349942526 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-28T07:49:15Z row to catalyst commit a2fdd4e1dca7067ae3fc7a76efde50851980fece Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-28T07:57:57Z getDate commit 0ed0fdc13ec043e16058128011428445a62c7581 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-01-28T08:15:51Z fix test data commit 096e20d5de068157910372a03a6face9edc829e6 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2015-02-03T06:29:01Z fix for mixed timezone --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72602223 Thanks for the quick fix. LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/4325#discussion_r23987594
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DateUtils.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.sql.Date
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * helper function to convert between Int value of days since 1970-01-01
and java.sql.Date
+ */
+object DateUtils {
--- End diff --
Should we mark it `private[spark]`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72607626 [Test build #26617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26617/consoleFull) for PR 4325 at commit [`096e20d`](https://github.com/apache/spark/commit/096e20d5de068157910372a03a6face9edc829e6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72607632 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26617/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4325#issuecomment-72601441 [Test build #26617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26617/consoleFull) for PR 4325 at commit [`096e20d`](https://github.com/apache/spark/commit/096e20d5de068157910372a03a6face9edc829e6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71795113 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26213/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71796516 [Test build #26216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26216/consoleFull) for PR 3732 at commit [`0ed0fdc`](https://github.com/apache/spark/commit/0ed0fdc13ec043e16058128011428445a62c7581). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71796124 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26215/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71796119 [Test build #26215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26215/consoleFull) for PR 3732 at commit [`a2fdd4e`](https://github.com/apache/spark/commit/a2fdd4e1dca7067ae3fc7a76efde50851980fece). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71797621 [Test build #26210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26210/consoleFull) for PR 3732 at commit [`f0005b1`](https://github.com/apache/spark/commit/f0005b166a705f7b1c52960b72c4ff29d010e5ff). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71797625 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26210/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71795022 [Test build #26215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26215/consoleFull) for PR 3732 at commit [`a2fdd4e`](https://github.com/apache/spark/commit/a2fdd4e1dca7067ae3fc7a76efde50851980fece). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71795107 [Test build #26213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26213/consoleFull) for PR 3732 at commit [`c37832b`](https://github.com/apache/spark/commit/c37832bc3a48493639b7a74d3277c11349942526). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71803990 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26216/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71803978 [Test build #26216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26216/consoleFull) for PR 3732 at commit [`0ed0fdc`](https://github.com/apache/spark/commit/0ed0fdc13ec043e16058128011428445a62c7581). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r23628599
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ---
@@ -252,7 +252,7 @@ trait Row extends Serializable {
*
* @throws ClassCastException when data type does not match.
*/
- def getDate(i: Int): java.sql.Date = apply(i).asInstanceOf[java.sql.Date]
+ def getDate(i: Int): java.sql.Date = DateUtils.toJavaDate(getInt(i))
--- End diff --
one thing - you probably want to do the conversion when we create the row,
like what we do for other types, instead of doing the conversion when it is
accessed. otherwise apply(i: Int) will return date.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71607951 Thanks for working on this! I took a quick look and it looks good. @yhuai can you do a more careful review? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71609833 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26148/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71609830 [Test build #26148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26148/consoleFull) for PR 3732 at commit [`024c9a6`](https://github.com/apache/spark/commit/024c9a6d37da9930b2c8d57cf4c01543ef0080e2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r23669844
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ---
@@ -252,7 +252,7 @@ trait Row extends Serializable {
*
* @throws ClassCastException when data type does not match.
*/
- def getDate(i: Int): java.sql.Date = apply(i).asInstanceOf[java.sql.Date]
+ def getDate(i: Int): java.sql.Date = DateUtils.toJavaDate(getInt(i))
--- End diff --
You can change the one in ScalaReflection.convertRowToScala to make it work
for both scala and java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r23670603
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ---
@@ -252,7 +252,7 @@ trait Row extends Serializable {
*
* @throws ClassCastException when data type does not match.
*/
- def getDate(i: Int): java.sql.Date = apply(i).asInstanceOf[java.sql.Date]
+ def getDate(i: Int): java.sql.Date = DateUtils.toJavaDate(getInt(i))
--- End diff --
Oh, sure...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71791681 [Test build #26210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26210/consoleFull) for PR 3732 at commit [`f0005b1`](https://github.com/apache/spark/commit/f0005b166a705f7b1c52960b72c4ff29d010e5ff). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r23669693
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ---
@@ -252,7 +252,7 @@ trait Row extends Serializable {
*
* @throws ClassCastException when data type does not match.
*/
- def getDate(i: Int): java.sql.Date = apply(i).asInstanceOf[java.sql.Date]
+ def getDate(i: Int): java.sql.Date = DateUtils.toJavaDate(getInt(i))
--- End diff --
Now I add the conversion in DataTypeConversion, which is only valid to java
class. For scala, We need to write
Row(DateUtils.fromJavaDate(...)). Is this OK with you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r23670522
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ---
@@ -252,7 +252,7 @@ trait Row extends Serializable {
*
* @throws ClassCastException when data type does not match.
*/
- def getDate(i: Int): java.sql.Date = apply(i).asInstanceOf[java.sql.Date]
+ def getDate(i: Int): java.sql.Date = DateUtils.toJavaDate(getInt(i))
--- End diff --
this line should be reverted since you changed
ScalaReflection.convertRowToScala right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71794073 [Test build #26213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26213/consoleFull) for PR 3732 at commit [`c37832b`](https://github.com/apache/spark/commit/c37832bc3a48493639b7a74d3277c11349942526). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r23670356
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ---
@@ -252,7 +252,7 @@ trait Row extends Serializable {
*
* @throws ClassCastException when data type does not match.
*/
- def getDate(i: Int): java.sql.Date = apply(i).asInstanceOf[java.sql.Date]
+ def getDate(i: Int): java.sql.Date = DateUtils.toJavaDate(getInt(i))
--- End diff --
Thanks, code updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71597172 [Test build #26146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26146/consoleFull) for PR 3732 at commit [`d6715fc`](https://github.com/apache/spark/commit/d6715fcac2a4c5a32a6b8c11f84567c675de1892). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71600826 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71602934 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26146/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71601166 [Test build #26148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26148/consoleFull) for PR 3732 at commit [`024c9a6`](https://github.com/apache/spark/commit/024c9a6d37da9930b2c8d57cf4c01543ef0080e2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-71602931 [Test build #26146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26146/consoleFull) for PR 3732 at commit [`d6715fc`](https://github.com/apache/spark/commit/d6715fcac2a4c5a32a6b8c11f84567c675de1892). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-70594597 Ping me when you are done updating this. Would be great to get into 1.3. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-70208781 Adrian - as we spoke offline, it would be simpler (for future datetime related features) to just represent the Date type as a primitive int internally, and convert to java.sql.Date when we give it back to the user. You can create a DateTimeUtils class to implement common functionalities such as conversion between strings and int date. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-70039985 [Test build #25595 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25595/consoleFull) for PR 3732 at commit [`4da81d4`](https://github.com/apache/spark/commit/4da81d4c9122528d62ac71f35087bfbca6c45f33). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-70043756 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25595/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-70043753 [Test build #25595 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25595/consoleFull) for PR 3732 at commit [`4da81d4`](https://github.com/apache/spark/commit/4da81d4c9122528d62ac71f35087bfbca6c45f33). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `final class Date extends Ordered[Date] with Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22990193
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ---
@@ -344,6 +345,11 @@ private[hive] trait HiveInspectors {
case _: JavaHiveDecimalObjectInspector =
(o: Any) =
HiveShim.createDecimal(o.asInstanceOf[Decimal].toBigDecimal.underlying())
+case _: JavaDateObjectInspector =
+ (o: Any) =
--- End diff --
That resulted from a bug elsewhere, now the bug has gone...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-70039420 According to https://docs.oracle.com/javase/7/docs/api/java/sql/Date.html To conform with the definition of SQL DATE, the millisecond values wrapped by a java.sql.Date instance must be 'normalized' by setting the hours, minutes, seconds, and milliseconds to zero in the particular time zone with which the instance is associated. While user can create whatever [[java.sql.Date]] instance to a certain milliseconds, this would lead to some potential problems. Also, the built-in comparing method will compare milliseconds, which is not right, as within the same day in different location, we have difference timezone. So we provide our own Date type here, 100% compatible with Hive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22955597
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Date.scala
---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Date to the given Int (days since 1970-01-01).
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Date to the given type.Date value.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ /**
+ * Get the Int value of days since 1970-01-01.
+ */
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ /**
+ * get the corresponding java.sql.Date value of this Date object.
+ */
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String =
Cast.threadLocalDateFormat.get.format(toJavaDate)
+
+ def toLong: Long = {
--- End diff --
rename this
toMillisSinceEpoch ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22955717
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Date.scala
---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
--- End diff --
we should put more information here, including why we can't just use
java.sql.Date and java.util.Date.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22955622
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Date.scala
---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Date to the given Int (days since 1970-01-01).
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Date to the given type.Date value.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ /**
+ * Get the Int value of days since 1970-01-01.
+ */
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ /**
+ * get the corresponding java.sql.Date value of this Date object.
+ */
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String =
Cast.threadLocalDateFormat.get.format(toJavaDate)
+
+ def toLong: Long = {
+val millisUtc = daysSinceEpoch.toLong * Date.MILLIS_PER_DAY
+millisUtc - Date.LOCAL_TIMEZONE.get().getOffset(millisUtc)
+ }
+
+ def toInt: Int = toLong.toInt
--- End diff --
again, please remove toInt, toShort, toByte.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22955738
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Date.scala
---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import java.util.{Calendar, TimeZone}
+
+import org.apache.spark.sql.catalyst.expressions.Cast
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Date to the given Int (days since 1970-01-01).
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Date to the given type.Date value.
+ */
+ def set(date: Date): Date = {
--- End diff --
why is this needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22955914
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ---
@@ -344,6 +345,11 @@ private[hive] trait HiveInspectors {
case _: JavaHiveDecimalObjectInspector =
(o: Any) =
HiveShim.createDecimal(o.asInstanceOf[Decimal].toBigDecimal.underlying())
+case _: JavaDateObjectInspector =
+ (o: Any) =
--- End diff --
when can we have two different types mixed here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69961857 Thanks for working on this. I made some comments removing methods exposed. Overall I favor minimalism in API design, because a simpler, smaller API is easier to learn. Not to mention that we cannot remove exposed APIs easily in the future. I'm still skeptical why we cannot use java.sql.Date. Can you elaborate? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69966856 To be clear, can you list all the problems when java.sql.Date is used in the context of Hive? Please list all the differences. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69882069 @adrian-wang https://github.com/apache/spark/pull/3958 has been merged. Can you bring this PR up to date? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69882219 [Test build #25525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25525/consoleFull) for PR 3732 at commit [`810c8c3`](https://github.com/apache/spark/commit/810c8c33582c778f32a381505ff72d858cfdb342). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69882319 Yes, I have just done the rebasing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69883249 [Test build #25525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25525/consoleFull) for PR 3732 at commit [`810c8c3`](https://github.com/apache/spark/commit/810c8c33582c778f32a381505ff72d858cfdb342). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `final class Date extends Ordered[Date] with Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69883254 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25525/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22921828
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
--- End diff --
the comment is wrong
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/3732#discussion_r22921778 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala --- @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the License); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.types.date --- End diff -- Actually we should probably put this in sql.types rather than the date package. I don't think we want to create a new package for each data type. https://issues.apache.org/jira/browse/SPARK-5245 Can you move it? Decimal was a bad example. Please do so if you want to submit a PR for moving Decimal too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22921859
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
--- End diff --
should add javadoc ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22921819
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
--- End diff --
is this needed at all?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22921862
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ def toJavaDate: java.sql.Date = {
--- End diff --
should add javadoc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/3732#discussion_r22922016 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala --- @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the License); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.types.date --- End diff -- That make sense, thanks! will do. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22921998
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String = toJavaDate.toString()
+
+ def toLong: Long = {
--- End diff --
Can we remove toLong, toInt, toShort, toByte? I'm not sure whether they are
useful. If they are, we can always add them in the future. Once they are added,
we cannot remove them easily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22922007
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String = toJavaDate.toString()
+
+ def toLong: Long = {
--- End diff --
It is also kind of consuing that toLong returns a different thing from
toInt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22922060
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/java/Row.scala
---
@@ -125,6 +126,7 @@ object Row {
case seq: scala.collection.Seq[_] =
JavaConversions.seqAsJavaList(seq.map(toJavaValue))
case decimal: BigDecimal = decimal.underlying()
+case date: Date = date.toJavaDate
--- End diff --
shouldn't java use the same Date in Scala?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22922206
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala ---
@@ -137,6 +138,8 @@ object EvaluatePython {
case (dec: BigDecimal, dt: DecimalType) = dec.underlying() //
Pyrolite can handle BigDecimal
+case (date: Date, DateType) = date.toJavaDate
--- End diff --
can't we convert our Date directly into Python?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22922237
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
---
@@ -253,7 +250,7 @@ case class Cast(child: Expression, dataType: DataType)
extends UnaryExpression w
// Converts Date to string according to Hive DateWritable convention
private[this] def dateToString(d: Date): String = {
--- End diff --
if Date already has a good toString implementaiton, maybe you can remove
this function entirely?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69884677 BTW one question I have - what is wrong with just using java.util.Date but trim the precision to date level? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69886900 We cannot decide what java.util.Date is, it is parent of both java.sql.Date and java.sql.Timestamp. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69887287 Then why can't we use java.sql.Date? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22924040
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String = toJavaDate.toString()
+
+ def toLong: Long = {
--- End diff --
This follows the cast rules of Hive. However, we can do this in Cast.scala
only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22924099
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/java/Row.scala
---
@@ -125,6 +126,7 @@ object Row {
case seq: scala.collection.Seq[_] =
JavaConversions.seqAsJavaList(seq.map(toJavaValue))
case decimal: BigDecimal = decimal.underlying()
+case date: Date = date.toJavaDate
--- End diff --
That's true, now it is a different story, from what it was when I wrote
these codes...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22924204
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String = toJavaDate.toString()
+
+ def toLong: Long = {
--- End diff --
Yea it makes more sense to do it in Cast.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3732#discussion_r22924226
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/date/Date.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types.date
+
+import java.util.{Calendar, TimeZone}
+
+/**
+ * A mutable implementation of java.sql.Date that holds an Int for days
since epoch, like Hive.
+ */
+final class Date extends Ordered[Date] with Serializable {
+ private var daysSinceEpoch: Int = 0
+
+ /**
+ * Set this Decimal to the given Long. Will have precision 20 and scale
0.
+ */
+ def set(days: Int): Date = {
+this.daysSinceEpoch = days
+this
+ }
+
+ /**
+ * Set this Decimal to the given Int. Will have precision 10 and scale 0.
+ */
+ def set(days: Long): Date = {
+this.daysSinceEpoch = days.toInt
+this
+ }
+
+ /**
+ * Set this Decimal to the given BigDecimal value, inheriting its
precision and scale.
+ */
+ def set(date: Date): Date = {
+this.daysSinceEpoch = date.daysSinceEpoch
+this
+ }
+
+ def toDays: Int = {
+this.daysSinceEpoch
+ }
+
+ def toJavaDate: java.sql.Date = {
+new java.sql.Date(toLong)
+ }
+
+ override def toString: String = toJavaDate.toString()
+
+ def toLong: Long = {
--- End diff --
BTW is this way of doing cast (toInt vs toLong having different semantics)
normal in other systems such as mysql, postgres?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69889826 We were using java.sql.Date originally, but it did not conform with hive... with different ordering definition(Though we can modify what we do when comparing, it would result in a worse performance), and also store it as an Int is helpful to save storage. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69890328 [Test build #25535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25535/consoleFull) for PR 3732 at commit [`2fea00c`](https://github.com/apache/spark/commit/2fea00c78c98ce7e3f72077144a9953bf4380718). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69890660 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25535/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69890654 [Test build #25535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25535/consoleFull) for PR 3732 at commit [`2fea00c`](https://github.com/apache/spark/commit/2fea00c78c98ce7e3f72077144a9953bf4380718). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69904110 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25543/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69904104 [Test build #25543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25543/consoleFull) for PR 3732 at commit [`aae27f4`](https://github.com/apache/spark/commit/aae27f450cf9dde42b0cf01b41d557a23c56ff8a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69896079 [Test build #25543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25543/consoleFull) for PR 3732 at commit [`aae27f4`](https://github.com/apache/spark/commit/aae27f450cf9dde42b0cf01b41d557a23c56ff8a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69487182 Actually can we merge this after we merge https://github.com/apache/spark/pull/3958 ? So @adrian-wang you will likely need to do a slightly larger rebase (should still be straightforward though). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69523451 Sure, I'll wait #3958 till it is merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-69479299 Hmm, okay. Thanks for doing the research! Can you fix the conflict so we can merge please? /cc @rxin more api considerations... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68815238 For oracle's doc about `java.sql.Date`, In https://docs.oracle.com/javase/6/docs/api/java/sql/Date.html To conform with the definition of SQL DATE, the millisecond values wrapped by a java.sql.Date instance must be 'normalized' by setting the hours, minutes, seconds, and milliseconds to zero in the particular time zone with which the instance is associated. For SQL-92 spec, http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt In page 111, 2) For a datetime type, Case: a) If DATE is specified, then the data type contains the date- time fields years, months, and days. The spec also clarifies that the definition is taken from ISO-8601[http://www.w3.org/TR/NOTE-datetime], in which it says, Complete date: -MM-DD (eg 1997-07-16) Hence, the time part of `java.sql.Date` is not necessary. I think that simply results from the inheritance of the same base type, namely `java.util.Date`, with `java.sql.Timestamp` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68618363 [Test build #25022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25022/consoleFull) for PR 3732 at commit [`2c2bda4`](https://github.com/apache/spark/commit/2c2bda455fb55b5b40bea0ef0a8a527c862d55a7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request:
https://github.com/apache/spark/pull/3732#issuecomment-68618384
Hi @marmbrus I have tried MySQL, and MySQL will treat cast('2015-01-03
18:25:04' as date) == cast('2015-01-03 18:29:02' as date), there is such a
precision lost.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68618399 The code seems conflict now, I have just rebased --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68620569 and MySQL doesn't allow to cast date back to timestamp type --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68390917 I'm a little confused about what semantics various systems are providing here. Does java.sql.Data provide greater precision that hive? If so, I'm not sure if we want to aim for the lowest common denominator. Can you investigate how other systems handle this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68243880 [Test build #24862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24862/consoleFull) for PR 3732 at commit [`3b4d5d8`](https://github.com/apache/spark/commit/3b4d5d80dc716a9fe2782115399a77f171d66cc7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
