[jira] [Commented] (SPARK-31508) string type compare with numberic cause data inaccurate
[ https://issues.apache.org/jira/browse/SPARK-31508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089663#comment-17089663 ] philipse commented on SPARK-31508: -- haha ,but normall it will be a little complex,we will migrate many hqls to sparksql，I suggest it will be better dealed with in the code. Can you help review the PR? > string type compare with numberic cause data inaccurate > --- > > Key: SPARK-31508 > URL: https://issues.apache.org/jira/browse/SPARK-31508 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 > Environment: hadoop2.7 > spark2.4.5 >Reporter: philipse >Priority: Major > Attachments: image-2020-04-22-20-00-09-821.png > > > Hi all > > Sparksql may should convert values to double if string type compare with > number type.the cases shows as below > 1, create table > create table test1(id string); > > 2,insert data into table > insert into test1 select 'avc'; > insert into test1 select '2'; > insert into test1 select '0a'; > insert into test1 select ''; > insert into test1 select > '22'; > 3.Let's check what's happening > select * from test_gf13871.test1 where id > 0 > the results shows below > *2* > ** > Really amazing,the big number 222...cannot be selected. > while when i check in hive,the 222...shows normal. > 4.try to explain the command,we may know what happened,if the data is big > enough than max_int_value，it will not selected,we may need to convert to > double instand. > !image-2020-04-21-18-49-58-850.png! > I wanna know if we have fixed or planned in 3.0 or later version.,please feel > free to give any advice, > > Many Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31508) string type compare with numberic cause data inaccurate
[ https://issues.apache.org/jira/browse/SPARK-31508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089598#comment-17089598 ] JinxinTang commented on SPARK-31508: such '2...' is far larger than big, I think we should not covert to long. > string type compare with numberic cause data inaccurate > --- > > Key: SPARK-31508 > URL: https://issues.apache.org/jira/browse/SPARK-31508 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 > Environment: hadoop2.7 > spark2.4.5 >Reporter: philipse >Priority: Major > > Hi all > > Sparksql may should convert values to double if string type compare with > number type.the cases shows as below > 1, create table > create table test1(id string); > > 2,insert data into table > insert into test1 select 'avc'; > insert into test1 select '2'; > insert into test1 select '0a'; > insert into test1 select ''; > insert into test1 select > '22'; > 3.Let's check what's happening > select * from test_gf13871.test1 where id > 0 > the results shows below > *2* > ** > Really amazing,the big number 222...cannot be selected. > while when i check in hive,the 222...shows normal. > 4.try to explain the command,we may know what happened,if the data is big > enough than max_int_value，it will not selected,we may need to convert to > double instand. > !image-2020-04-21-18-49-58-850.png! > I wanna know if we have fixed or planned in 3.0 or later version.,please feel > free to give any advice, > > Many Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31508) string type compare with numberic cause data inaccurate
[ https://issues.apache.org/jira/browse/SPARK-31508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089596#comment-17089596 ] JinxinTang commented on SPARK-31508: for example： select * from default.test1 where id > 3D add 'D' after number 3,this will not convert to long and int may alse work, have fun~ > string type compare with numberic cause data inaccurate > --- > > Key: SPARK-31508 > URL: https://issues.apache.org/jira/browse/SPARK-31508 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 > Environment: hadoop2.7 > spark2.4.5 >Reporter: philipse >Priority: Major > > Hi all > > Sparksql may should convert values to double if string type compare with > number type.the cases shows as below > 1, create table > create table test1(id string); > > 2,insert data into table > insert into test1 select 'avc'; > insert into test1 select '2'; > insert into test1 select '0a'; > insert into test1 select ''; > insert into test1 select > '22'; > 3.Let's check what's happening > select * from test_gf13871.test1 where id > 0 > the results shows below > *2* > ** > Really amazing,the big number 222...cannot be selected. > while when i check in hive,the 222...shows normal. > 4.try to explain the command,we may know what happened,if the data is big > enough than max_int_value，it will not selected,we may need to convert to > double instand. > !image-2020-04-21-18-49-58-850.png! > I wanna know if we have fixed or planned in 3.0 or later version.,please feel > free to give any advice, > > Many Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31508) string type compare with numberic cause data inaccurate
[ https://issues.apache.org/jira/browse/SPARK-31508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089583#comment-17089583 ] JinxinTang commented on SPARK-31508: You are right,We can see from code gen in branch 2.4 which will convet use org.apache.spark.unsafe.types.UTF8String#toInt method > string type compare with numberic cause data inaccurate > --- > > Key: SPARK-31508 > URL: https://issues.apache.org/jira/browse/SPARK-31508 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 > Environment: hadoop2.7 > spark2.4.5 >Reporter: philipse >Priority: Major > > Hi all > > Sparksql may should convert values to double if string type compare with > number type.the cases shows as below > 1, create table > create table test1(id string); > > 2,insert data into table > insert into test1 select 'avc'; > insert into test1 select '2'; > insert into test1 select '0a'; > insert into test1 select ''; > insert into test1 select > '22'; > 3.Let's check what's happening > select * from test_gf13871.test1 where id > 0 > the results shows below > *2* > ** > Really amazing,the big number 222...cannot be selected. > while when i check in hive,the 222...shows normal. > 4.try to explain the command,we may know what happened,if the data is big > enough than max_int_value，it will not selected,we may need to convert to > double instand. > !image-2020-04-21-18-49-58-850.png! > I wanna know if we have fixed or planned in 3.0 or later version.,please feel > free to give any advice, > > Many Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31508) string type compare with numberic cause data inaccurate
[ https://issues.apache.org/jira/browse/SPARK-31508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089449#comment-17089449 ] JinxinTang commented on SPARK-31508: select id from test1 where id>'0'; may be seems more normal. Thanks for your issue,I will inspect the code too. > string type compare with numberic cause data inaccurate > --- > > Key: SPARK-31508 > URL: https://issues.apache.org/jira/browse/SPARK-31508 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5 > Environment: hadoop2.7 > spark2.4.5 >Reporter: philipse >Priority: Major > > Hi all > > Sparksql may should convert values to double if string type compare with > number type.the cases shows as below > 1, create table > create table test1(id string); > > 2,insert data into table > insert into test1 select 'avc'; > insert into test1 select '2'; > insert into test1 select '0a'; > insert into test1 select ''; > insert into test1 select > '22'; > 3.Let's check what's happening > select * from test_gf13871.test1 where id > 0 > the results shows below > *2* > ** > Really amazing,the big number 222...cannot be selected. > while when i check in hive,the 222...shows normal. > 4.try to explain the command,we may know what happened,if the data is big > enough than max_int_value，it will not selected,we may need to convert to > double instand. > !image-2020-04-21-18-49-58-850.png! > I wanna know if we have fixed or planned in 3.0 or later version.,please feel > free to give any advice, > > Many Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
